Pages

Friday, September 22, 2017

Fedora 26 + Virtualbox 5.1 + kenel 4.12

Upgrading your virtual machine (VM) environment can sometimes lead to unexpected issues. A common problem users might encounter after upgrading VirtualBox is the VM failing to start. In this post, we'll walk through a specific error and provide a step-by-step guide to resolve it, ensuring your virtual environment gets back up and running smoothly.

Understanding the Error

Upon attempting to start a VM after an upgrade, you might encounter an error in your logs similar to this:

/tmp/vbox.0/r0drv/linux/memuserkernel-r0drv-linux.o: warning: objtool: .fixup: unexpected end of section if [ "-pg" = "-pg" ]; then if [ /tmp/vbox.0/r0drv/linux/memuserkernel-r0drv-linux.o != "scripts/mod/empty.o" ]; then ./scripts/recordmcount "/tmp/vbox.0/r0drv/linux/memuserkernel-r0drv-linux.o"; fi; fi; make[1]: *** [Makefile:1519: _module_/tmp/vbox.0] Error 2 make: *** [Makefile:304: vboxdrv] Error 2

This error typically indicates a problem with the VirtualBox kernel modules not compiling or loading correctly due to incompatibilities or issues within the system.

Step-by-Step Solution

Fear not, as this issue can often be resolved by applying a patch to the VirtualBox source. Here's how you can fix it:

1. Change to the VirtualBox Source Directory:

Navigate to the directory where VirtualBox sources are stored:

cd /usr/share/virtualbox/src

2. Obtain the Necessary Patch:

Download the patch designed to fix the issue:

sudo wget http://paste.siduction.org/20170629003423

3. Apply the Patch:

Apply the downloaded patch to the VirtualBox source:

sudo patch -Np0 < 20170629003423

4. Reconfigure VirtualBox:

After applying the patch, you need to reconfigure VirtualBox to make sure it recognizes the changes:

/sbin/vboxconfig

Post-Solution Tips:

Have Fun! Now that you've resolved the issue, your VMs should start as expected. Dive back into your virtual environment and continue your work or play.

Stay Updated: Keep your system and VirtualBox updated to avoid similar issues in the future. Developers regularly release patches and updates to address known bugs and compatibility issues.

Seek Community Help: If you encounter further issues or the problem persists, don't hesitate to seek help from the VirtualBox community forums or check out other user experiences for additional insights.

Conclusion

Encountering errors after a system or software upgrade can be frustrating, but with the right approach and resources, most issues can be resolved. By understanding the error, carefully following the provided steps, and engaging with the community, you can overcome challenges and enjoy a seamless virtualization experience with VirtualBox. Keep exploring, learning, and sharing your knowledge with others!

Friday, September 8, 2017

Minio Running as Service

Minio is a distributed object storage server, similar to Amazon S3, that allows you to store and access large amounts of data. Since the service is running on different hosts, it is important to have a shared storage mechanism so that the data is synchronized across all nodes. To achieve this, a bind mount is used to mount a directory on the host machine to the Minio server container, allowing it to read and write data to the directory. Additionally, two Docker secrets are created for access and secret keys to authenticate and authorize access to the Minio server. Finally, the service is created with the docker service create command, specifying the name of the service, the port to publish, the constraint to run the service only on a manager node, the bind mount for data synchronization, and the two Docker secrets for authentication. The minio/minio image is used to run the Minio server, and the /data directory is specified as the location to store data.


echo "AKIAIOSFODNN7EXAMPLE" | docker secret create access_key -
echo "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY" | docker secret create secret_key -

docker service create --name="minio-service" --publish 9000:9000   --constraint 'node.role == manager' --mount type=bind,src=/mnt/minio/,dst=/data --secret="access_key" --secret="secret_key" minio/minio server /data

Wednesday, September 6, 2017

Minio: S3 Compatible Stoage in Docker

Minio is a distributed object storage server that is designed to be scalable and highly available. It is built for cloud-native applications and DevOps. Minio provides Amazon S3 compatible API for cloud-native applications to store and retrieve data. It is open-source and can be deployed on-premise, on the cloud or on Kubernetes.

The command docker pull minio/minio pulls the Minio image from Docker Hub. The command docker run -p 9000:9000 minio/minio server /data runs a Minio container with port forwarding from the host to the container for the Minio web interface. The /data parameter specifies the path to the data directory that will be used to store the data on the container's file system.

**We need to have the docker env up and running.

docker pull minio/minio
docker run -p 9000:9000 minio/minio server /data


After running this command, you can access the Minio web interface by navigating to http://localhost:9000 in your web browser.






Thursday, August 17, 2017

Inceass the Root Disk Size for Centos in Aws

Issue: Root Partition not scaled after EBS is resized.

Growpart called by cloud-init only works for kernels >3.8. Only newer kernels support changing the partition size of a mounted partition. When using an older kernel the resizing of the root partition happens in the initrd stage before the root partition is mounted and the subsequent cloud-init growpart run is a no-op.


#lsblk
NAME    MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
xvda    202:0    0  30G  0 disk
└─xvda1 202:1    0   8G  0 part /
Perform the following command as root:

# yum install cloud-utils-growpart

# growpart /dev/xvda 1

# reboot
After the reboot:

# lsblk
NAME    MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
xvda    202:0    0  30G  0 disk
└─xvda1 202:1    0  30G  0 part /

Sunday, August 13, 2017

Qubole : Load Multiple tables to Qubole Hive table from a Data Store

API call to Load Multiple tables from a Qubole Data Store to Hive table. 


[rahul@local qubole]$ cat /databasescript 
#!/bin/bash

#Qubole API Key
AUTH="***********"
#Database Name
DB_NAME="***********"
#Host Name
DB_HOST="***********"
#User Name
DB_USER="***********
#Password 
DB_PASS='***********'

echo $DB_PASS


## request table import from tap;
function tableImport() {

request_body=$(cat <<EOF
{
   "command_type":"DbImportCommand",
   "mode":"1",
   "hive_serde":"orc",
   "hive_table":"<HIVE TABLE NAME>.$1",
   "dbtap_id":"$2",
   "db_table":"$1",
   "db_parallelism":"1",
   "use_customer_cluster":"1",
   "customer_cluster_label":"Qubole_Data_Import",
   "tags":[" Data"]
}
EOF
)

echo $request_body
   curl -X POST \
-H "X-AUTH-TOKEN: $AUTH" \
-H "Content-Type:application/json" \
-d "$request_body" https://api.qubole.com/api/v1.2/commands/
}

##register database with tap
request_body=$(cat <<EOF
{
  "db_name":"$DB_NAME",
  "db_host":"$DB_HOST",
  "db_user":"$DB_USER",
  "db_passwd":"$DB_PASS",
  "db_type":"sqlserver",
  "db_location":"on-premise",
  "gateway_ip": "***********",
  "gateway_port": "***********",
  "gateway_username": "***********",
  "gateway_private_key": "***********"}

EOF
)

echo $KEY
ID=$(curl -s -X POST \
-H "X-AUTH-TOKEN: $AUTH" \
-H "Content-Type:application/json" \
-d "$request_body" https://api.qubole.com/api/v1.2/db_taps/ | jq .id)

#get the tables and call import
curl -s -H "X-AUTH-TOKEN: $AUTH" \
     -H "Content-Type:application/json" \
     https://api.qubole.com/api/v1.2/db_taps/$ID/tables | jq -r .[] | while read x; do  tableImport $x $ID; done

# can't delete the tap at the end unless we continuously poll for no active jobs;
STATUS="null"

while [ "$STATUS" = "null" ]
do
STATUS=$(curl  -s -X DELETE \
 -H "X-AUTH-TOKEN: $AUTH" \
 -H "Content-Type:application/json" \
 https://api.qubole.com/api/v1.2/db_taps/$ID | jq .status)
echo -n "."
sleep 5
done

Thursday, July 6, 2017

GrayLog Configuration Error : Please verify that the server is healthy and working correctly.

First, we need to Make sure the Elastic Seach is running fine.

Following was the configuration
cluster.name: graylog
network.host: 127.0.0.1

Then Make sure the Entry in the graylog for following attributes is correct.



rest_listen_uri = http://0.0.0.0:9000/api
web_listen_uri = http://0.0.0.0:9000/
rest_transport_uri = http://192.168.0.66:9000/api
web_endpoint_uri = http://192.168.0.66:9000/api

** In Aws/Azure Make sure we give the Server's Public IP or the Load balancers IP.

Friday, June 30, 2017

Reduce TIME_WAIT socket connections

We will reduce the Time_wait by tweaking the Sysctl to time out at a certain time and reuse that socket.

List the no of time_waits and Established Connections

>>netstat -nat | awk '{print $6}' | sort | uniq -c | sort -n

cat /proc/sys/net/ipv4/tcp_fin_timeout
cat /proc/sys/net/ipv4/tcp_tw_recycle
cat /proc/sys/net/ipv4/tcp_tw_reuse

If you have default settings, you’ll probably see values of 60, 0 and 0. Let’s change those values to 60, 1, 1.

Now, edit the /etc/sysctl.conf with your favorite editor and add these lines to the end of it (or edit the values you have in yours if they exist already):


# Decrease TIME_WAIT seconds
net.ipv4.tcp_fin_timeout = 30

# Recycle and Reuse TIME_WAIT sockets faster
net.ipv4.tcp_tw_recycle = 1
net.ipv4.tcp_tw_reuse = 1

Sysctl -p

netstat -nat | awk '{print $6}' | sort | uniq -c | sort -n