Delete a VM in an Error State in Openstack

Delete a VM in an Error State

mysql -u$MYSQL_USER -p$MYSQL_PASSWD -h$MYSQL_HOST -e 'USE nova; DELETE FROM security_group_instance_association WHERE instance_id IN (SELECT id FROM instances WHERE vm_state = "error");'
mysql -u$MYSQL_USER -p$MYSQL_PASSWD -h$MYSQL_HOST -e 'USE nova; DELETE FROM block_device_mapping WHERE instance_id IN (SELECT id FROM instances WHERE vm_state = "error");'
mysql -u$MYSQL_USER -p$MYSQL_PASSWD -h$MYSQL_HOST -e 'USE nova; DELETE FROM instance_info_caches WHERE instance_id IN (SELECT uuid FROM instances WHERE vm_state = "error");'
mysql -u$MYSQL_USER -p$MYSQL_PASSWD -h$MYSQL_HOST -e 'USE nova; UPDATE fixed_ips SET allocated = 0 WHERE instance_id IN (SELECT id FROM instances WHERE vm_state = "error");'
mysql -u$MYSQL_USER -p$MYSQL_PASSWD -h$MYSQL_HOST -e 'USE nova; DELETE FROM instances WHERE vm_state = "error";'

Resetting the Instance state once it’s in a stuck state in Openstack

Resetting the Instance state once it’s in a stuck state

$ nova list
+--------------------------------------+----------------+---------+------------------------------------------.  -------------+-------------+----------------------------------+
| ID                                                                 | Name         | Status        |   Task State  | Power State | Networks                                      |
+--------------------------------------+----------------+---------+-------------------------------------------  -------------+-------------+----------------------------------+
| ca9496e9-0bd2-4734-9cf9-eb4e264628f7 | www            | SHUTOFF | powering-on | Shutdown    | fsf-lan=10.0.3.18, 93.20.168.177  |
+--------------------------------------+----------------+---------+----------------------------------------------------------+-------------+----------------------------------+

   
Setting the fields for the instance directly in the database will allow operations on the instance (nova start or nova volume-detach for instance):

$ mysql -e "update instances set task_state = NULL, \
           vm_state = 'stopped', \
           power_state = 4 \
           where deleted = 0 and hostname = 'www' and \
           uuid = 'ca9496e9-0bd2-4734-9cf9-eb4e264628f7'" nova

Removing a Cinder volume from the MYSQL in Openstack

Get the Cinder Volume’s ID from Cinder List and use it in other commands

>>cinder list;
$> nova volume-detach testvm aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee
$> cinder delete vol-00000001
Or
In mysql Server
mysql
use cidner;
SELECT id,status,attach_status,mountpoint,instance_uuid from volumes;
UPDATE volumes SET status="available", attach_status="detached", mountpoint=NULL, instance_uuid=NULL WHERE id="57b32eaf-7b71-49bf-a8fd-4115567a6cda";
SELECT id,status,attach_status,mountpoint,instance_uuid from volumes;
bye

Controlling Shell-Fork-Bombing in linux User's

ulimit: Provides control over the resources available to the  shell  and to  processes started by it, on systems that allow such control. The -H and -S options specify that the hard or soft limit is set for  the  given resource.  A hard limit cannot be increased by a non-root user once it is set; a soft limit may be  increased  up to  the value of the hard limit.  If neither -H nor -S is speci- fied, both the soft and hard limits are set.  The value of limit can be a number in the unit specified for the resource or one of the special values hard, soft, or unlimited, which stand for the current  hard  limit,  the  current  soft  limit,  and no limit, respectively.  If limit is omitted, the  current  value  of  the soft  limit  of the resource is printed, unless the -H option is given.  When more than one resource is specified, the limit name and unit are printed before the value. 

 Other options are inter-preted as follows:
-a     All current limits are reported
-b     The maximum socket buffer size
-c     The maximum size of core files created
-d     The maximum size of a process’s data segment
-e     The maximum scheduling priority ("nice")
-f     The maximum size of files written by the  shell  and  its children
-i     The maximum number of pending signals
-l     The maximum size that may be locked into memory
-m     The  maximum resident set size (many systems do not honor this limit)
-n     The maximum number of open file descriptors (most systems do not allow this value to be set)
-p     The pipe size in 512-byte blocks (this may not be set)
-q     The maximum number of bytes in POSIX message queues
-r     The maximum real-time scheduling priority
-s     The maximum stack size
-t     The maximum amount of cpu time in seconds
-u     The maximum number of processes available to a single user
-v     The maximum amount of virtual memory available to the shell
-x     The maximum number of file locks
-T     The maximum number of threads

You can add the following to bashrc file for protecting the local shell users from fork bomb attack.

#unlimit so we can run the whoami
ulimit -n 4096 -u 14335 -m unlimited -d unlimited -s 8192 -c 1000000 -v unlimited 2>/dev/null

LIMITUSER=$USER
if [ -e "/usr/bin/whoami" ]; then
        LIMITUSER=`/usr/bin/whoami`
fi
if [ "$LIMITUSER" != "root" ]; then
        ulimit -n 100 -u 35 -m 200000 -d 200000 -s 8192 -c 200000 -v unlimited 2>/dev/null
else
        ulimit -n 4096 -u 14335 -m unlimited -d unlimited -s 8192 -c 1000000 -v unlimited 2>/dev/null
fi

Nova issue with deleting Vm

We can delete the nova instance which are in error/deleting state by following the steps mentioned below.

Get the instance ID from command “nova list ” and reset the state using “nova reset-state <ID>” command

nova list

[root@controller1 ~]# nova list
+--------------------------------------+-----------------+--------+------------+-------------+----------------------------------------------+
| ID                                   | Name            | Status | Task State | Power State | Networks                                     |
+--------------------------------------+-----------------+--------+------------+-------------+----------------------------------------------+
| 0ccfa148-97de-4be9-b85d-4283037746b1 | Ha-Porxy-F5     | ACTIVE | -          | Running     | Trusted-Internal-NIC=10.0.0.157, 10.1.15.140 |
| e61a759d-528c-423b-bb24-dcf7e3a5618e | Ha-Porxy-Mysql  | ACTIVE | -          | Running     | Trusted-Internal-NIC=10.0.0.158, 10.1.15.139 |

nova reset-state 0ccfa148-97de-4be9-b85d-4283037746b1


You can also use the --active parameter to force the instance back to an active state instead of an error state. For example:

$ nova reset-state --active c6bbbf26-b40a-47e7-8d5c-eb17bf65c485

Neutron check Commands

To list all the virtual routers and dhcp server
>> ip netns ls

[root@neutronww1 ~]# ip netns
qrouter-641ca25a-7832-4818-b7ww5b-559c8f75ba5c
qdhcp-71ed8a34-a2d5-4d84-9dww47-e5e107dd8d7e
qdhcp-35a33370-d641-4dca-9wwdab-4ac2d9ffc7c6
qdhcp-e0b51c09-57dd-4b3a-a1ww1f-c83903d52e4d
qrouter-3b7f1bcc-0c95-47d7-a2ww17-9fe2aef7f0c0
qrouter-ac17d3c5-9bf4-4788-b2www6c-ca8aa249613b
[root@neutron1 ~]#

Here the virtual routers are once which start with extension qrouter and dhcp server are those which start with extension qdhcp.


To get more details about the virtual routers and dhcp server we can use following command

>>[root@n1 ~]# ip netns exec <virtual router/dhcp server IP from ip netns command> <network command>

Examples

ip netns exec qrouter-641ca25a-7832-4818-b7ww5b-559c8f75ba5c ip a
ip netns exec qrouter-641ca25a-7832-4818-b7www5b-559c8f75ba5c ifconfig
ip netns exec qrouter-641ca25a-7832-4818-b7www5b-559c8f75ba5c route -n
ip netns exec qrouter-641ca25a-7832-4818-bwww75b-559c8f75ba5c ping

In the above examples we use the commands like “ip a”, “ifconfig” and “route” to list different parameters of the virtual router and dhcp server.  We can use the route command to add up more routing rules if needed.  There we can use the commands like ip, ifconfig, route, ping etc just as we use it on the physical system. To tweak/troubleshoot the entire system. 

Bridge Interface shown down in Horizon Dashboard @ Openstack

sudo ovs-vsctl br-get-external-id br-ex returns nothing, and so br-ex is excluded from the list of ancillary bridges and so the gateway port always shows as DOWN.
A workaround is to set the bridge-id to br-ex and restart the L2 agent:

ovs-vsctl br-set-external-id br-ex bridge-id br-ex
ovs-vsctl br-set-external-id br-ex-2 bridge-id br-ex-2