Python Server side comments added and readme updated

WirelessX-Python-Server/README.md

+# WIRELESS X
+
+
+
+## Installation Setup
+
+    1. Make sure that you are in the Wireless-X source directory
+
+    2. Grant the permission to execute install.sh installation script using the following command:
+            $ sudo chmod a+x install.sh
+
+    3. Execute the install.sh script to install the necessary dependencies using the following command:
+            $ sudo ./install.sh
+
+## Running the application (Strictly follow the below order to run it successfully):
+
+    1. Run the Wireless-X server using the following command:
+            $ python3 Wireless-X_server.py
+    
+    2. Enter your linux system password (the same password you enter while executing a command as "sudo"). This is required in order to set up the virtual webcam device.               
+
+    3. Application Installation and Setup on Android Smartphone:
+
+        a. Install the Wireless-X apk on Android smartphone and give required permissions.
+
+        b. Now enter the IP address of the Laptop/ Desktop on which the server is hosted. You can type the below command in terminal to get host IP:
+                $ hostname -I | awk '{print $1}'
+        
+        c. Click on Test Button to test the connection of smartphone with the server. If failed, Recheck if you have entered correct IP address of Laptop/ Desktop (on which the server is running).
+
+        d. After successfull connection, you would be able to control mouse, keyboard of laptop and use smartphones camera as webcam for the laptop/Desktop.
+
+        e. Now you would be able to use this virtual webcam device on chrome for video conferincing. (Tested on chrome for MS Teams and Google Meet).
+
+        f. (Optional) Inorder to test if camera frames are received to the Laptop/ Desktop, use the below command while (Note: camera option should be turned on in the 
+            Wireless-X apk on Android):
+                $ ffplay /dev/video20 
+
+
+#### *Extra (Inorder to remove v42loopback devices, use below command):
+            
+            $ sudo modprobe -r v4l2loopback
+
+
+## Steps for Debugging (If python code doesn't run after above commands):
+
+    1. Check if your virtual device is created
+
+        $ ls /dev | grep -P '^video\d+$'
+        OR
+        $ v4l2-ctl --list-devices    # TO List the virtual devices in detail
+
+            Output should look somewhat like this:
+
+                Wireless-X Camera (platform:v4l2loopback-000):
+                    /dev/video20
+
+                Webcam C170: Webcam C170 (usb-0000:00:1a.0-1.2):
+                    /dev/video0
+                    /dev/video1
+
+
+    2. Inorder to test if virtual device is working:
+        Copy the sample code from  https://github.com/jremmons/pyfakewebcam page and save it as python file and run it.
+            
+            $ python3 demo.py
+
+        If everything worked correctly, no error should be displayed and terminal should be blank.
+        Now, Open another terminal and test if virtual device output is being display by entering below command:
+            
+            $ ffplay /dev/video20
+        
+        Note: ffplay
+
+
+
+## References
+
+OpenCV Reference:
+
+    https://cmsdk.com/java/android-opencv-tcp-video-streaming.html
+
+
+v4l2loopback References:
+
+    https://github.com/jremmons/pyfakewebcam
+    https://github.com/jremmons/pyfakewebcam/issues/5
+    https://github.com/umlaeute/v4l2loopback#DKMS
+
+
+Android Reference:
+
+    https://stackoverflow.com/questions/23024831/android-shared-preferences-for-creating-one-time-activity-example
\ No newline at end of file

WirelessX-Python-Server/Wireless-X_server.py

+##
+# @file
+# This includes the code for the server-side of Wireless-X
+#
+# The server running on laptop or PC is responsible for receiving the actions performed by user on the
+# Wireless-X android app as well as receiving the camera frames of the user's smartphone (if the user has turned)
+# on the camera). Such actions are transmitted to the server in encoded form, the server decodes the received
+# message and instructs the laptop or PC to perform the action described in that message.
+##
+
 import socket
 import time
 import threading
@@ -6,36 +16,72 @@ import cv2
 import numpy as np
 import binascii
 import pyfakewebcam
+import subprocess
 from pynput.keyboard import Key, Controller as KeyboardController
 from pynput.mouse import Button, Controller as MouseController
 
-print('---------Wireless-X Server------')
+## Creates a virtual camera on the laptop/PC
+virtualCamera = subprocess.run(["sudo", "modprobe", "v4l2loopback", "devices=1", "video_nr=20", "card_label='Wireless-X Camera'", "exclusive_caps=1"])
+print("Virtual camera status:", virtualCamera.returncode)
 
+print('----------Wireless-X Server----------')
+print('Press Ctrl+C to terminate the server')
 
+## @var width
+# Stores the width of the screen
+## @var height
+# Stores the height of the screen
 width, height = autopy.screen.size()
 print("width: "+str(width)+" height: "+str(height))
+
+## @var curr_x
+# Stores the x-coordinate of the current mouse location
+## @var curr_y
+# Stores the y-coordinate of the current mouse location
 curr_x, curr_y = autopy.mouse.location()
+
+## Stores the mid of x-coordinate of current mouse location
 remote_x = curr_x/2
+## Stores the mid of y-coordinate of current mouse location
 remote_y = curr_y/2
+
+## Socket used for receiving keyboard and mouse related actions
 s=''
+
+## Socket used for receiving the camera frames of user's smartphone
 cameraSocket=''
+
+## Width of the camera frame
 img_width = 720
+
+## Height of the camera frame
 img_height = 480
+
+## Virtual webcam device
 camera = pyfakewebcam.FakeWebcam('/dev/video20', img_width, img_height)
+
+## The camera and keyboard-mouse sockets receive user requests until this variable is set to 'True'
 thread_run = True
 
+## Initializing the KeyboardController object
 keyboard = KeyboardController()
-mouse = MouseController()
 
-mouse_speed =2
+## Initializing the MouseController object 
+mouse = MouseController()
 
+## Speed of mouse movement
+mouse_speed = 2
 
-screenshot_count=0
+## Screenshot counter to keep track of screenshots
+screenshot_count = 0
 
-# binding the socket
+##
+# @brief This function establishes two sockets for receiving camera frames as well as mouse and keyboard actions 
+##
 def bind_sockets():
+    """This function creates a camera socket which is responsible for receiving the camera frames, and it also
+    creates another socket which is responsible for receiving the keyboard and mouse frames."""
     try:
-
         global s, cameraSocket
         
         cameraSocket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
@@ -56,51 +102,24 @@ def bind_sockets():
         print("Keyboard and Mouse Listening port: "+ str(key_port))
         s.listen(10)
 
-
     except socket.error as msg:
         print("Socket Binding error: "+str(msg))
         time.sleep(5) # Time to sleep before reattempting the bind connection
         bind_sockets()
 
 
-
-
-'''
-Testing Zone
-
-
-
-
-
-jsjsjjjsu
-rajeesh n  rajneesh katkam
-~12838
-
-    qwertyuiop[]
-    ASDFasdfghjkl;'
-    zxcvbnm,           ./
-zxcvbnzxcrvb<<<<<<<>>>>>>><<<<<~~~~~u@(@@*@wjwjsjiwi@u<><<<<<<<<<<<<<<<<>>>>>>>>>>>>>>)
-
-Te84848  QWEqweUTREYYUWYWqtwywywuiiwywyRTWWYYUUUYETUIOIwwertyuiop[][
-special_key_list = ["F1",  "F2",  "F3",  "F4",  "F5",  "F6",  "F7",  "F8",  "F9",  "F10",  "F11",  "F12",  "ALT",  "BACKSPACE",  "CAPS_LOCK",  "CONTROL",  "DELETE",  "DOWN_ARROW",  "END",  "ESCAPE",  "HOME",  "LEFT_ARROW",  "META",  "PAGE_DOWN",  "PAGE_UP",  "RETURN",  "RIGHT_ARROW",  "SHIFT",  "SPACE",  "UP_ARROW"]
-
-special_key_android_list = ["F1",  "F2",  "F3",  "F4",  "F5",  "F6",  "F7",  "F8",  "F9",  "F10",  "F11",  "F12",  "Alt",  "Backspace",  "Caps\nLock",  "Ctrl",  "Delete",  "↓",  "End",  "Esc",  "Home",  "←",  "META",  "Page Down",  "Page Up",  "Enter",  "→",  "Shift",  "Space",  "↑"]
-
-
-special_key_android_dictionary = {
-
-    "Alt":"ALT",  "Caps\nLock":"CAPS_LOCK",  "Ctrl":"CONTROL",
-
-     "META":"META",
-
-    "Enter":"RETURN", "Shift":"SHIFT", "Space":"SPACE", "Tab":"Tab"}
-
-'''
-
-
+## Maps the keys in keyboard layout to the actual keyboard keys
 special_key_android_dictionary = {"F1": "F1", "F2":"F2", "F3":"F3", "F4":"F4", "F5":"F5", "F6":"F6", "F7":"F7", "F8":"F8", "F9":"F9", "F10":"F10", "F11":"F11", "F12":"F12", "Alt":"ALT", "Backspace":"BACKSPACE",  "Caps\nLock":"CAPS_LOCK",  "Ctrl":"CONTROL", "Delete":"DELETE",  "↓":"DOWN_ARROW",  "End":"END", "Esc":"ESCAPE", "Home":"HOME", "←":"LEFT_ARROW", "META":"META", "Page Down":"PAGE_DOWN", "Page Up":"PAGE_UP", "Enter":"RETURN", "→":"RIGHT_ARROW", "Shift":"SHIFT", "Space":"SPACE", "↑":"UP_ARROW", "Tab":"Tab"}
 
+##
+# @brief This function decodes the received mouse and keyboard actions and acts accordingly 
+##
 def mouse_keyboard_connections():
+    """This function checks if the message received corresponds to a mouse action (left-click, scroll, etc.) or a keyboard action 
+    (such as keypress). It then instructs the laptop to perform these actions, using the 'autopy' and 'pynput' libraries. Some special 
+    characters in keyboard are not supported by 'autopy' library, so the actions corresponding to these special characters are performed
+    by the 'pynput' library, other key actions are handled by the 'autopy' library. In case of mouse, the 'autopy' library was not that 
+    efficient, so we used the 'pynput' library."""
     global thread_run, curr_x, curr_y, remote_x, remote_y
     global screenshot_count
     thread_run=True
@@ -120,31 +139,14 @@ def mouse_keyboard_connections():
                 xy=peer_response.split(',')
                 if len(xy)==2:
                     mouse.position = (float(xy[0])*width, float(xy[1])*height)
-                    #mouse.position = (float(xy[0])*mouse_speed, float(xy[1])*mouse_speed)
-
-                    #mouse.move((float(xy[0])-0.5)*10, (float(xy[0])-0.5)*10)
-
-                    #print(xy[1]+","+xy[0], end="\t")
-                    #print(str(width), str(height), end="\t")
-                    #remote_x= float(max(min(float(xy[0]),1),0)*width) # + prev_x
-                    #remote_y=  float(max(0,min(float(xy[1]),1),0)*height) # + prev_y
-                    #autopy.mouse.move(remote_x, remote_y)
-
-                    #mouse.position = (remote_x, remote_y)
-                    
-
 
             elif "!#Keyboard#!" in peer_response:
                 peer_response=peer_response.split("!#Keyboard#!")[1]
-                #print("Key:"+ peer_response)
-                #autopy.key.tap(autopy.key.Code.TAB, [autopy.key.Modifier.META])
-                #autopy.key.tap(autopy.key.Code.RETURN)
                 if peer_response in special_key_android_dictionary.keys():
                     
                     print("Special: "+ special_key_android_dictionary[peer_response])
 
                     if peer_response == "Backspace":
-                        #pyautogui.press('backspace')
                         keyboard.press(Key.backspace)
                         keyboard.release(Key.backspace)
                     
@@ -160,7 +162,6 @@ def mouse_keyboard_connections():
 
                 elif peer_response == "Print\nScreen":
                     print("Entered Print Screen: "+ str(screenshot_count))
-                    #autopy.bitmap.capture_screen().save(str("screenshot.png"))
                     autopy.bitmap.capture_screen().save(str("screenshot_"+str(screenshot_count)+".png"))
                     screenshot_count+=1
 
@@ -190,25 +191,20 @@ def mouse_keyboard_connections():
                 else:
                     mouse.scroll(0, -2)  # Scroll Down
                     
-
-
             elif "!#Test#!" in peer_response:
-                #print(peer_response)p
-                conn.send(bytes("Success "+str(int(width))+" "+str(int(height)),"utf-8"))
-
-            # else:
-            #     print(peer_response)
+                conn.send(bytes("Success "+str(int(width/mouse_speed))+" "+str(int(height/mouse_speed)),"utf-8"))
 
             conn.close()
 
-        except Exception as msg :
-            # if "timed out" not in str(msg):
-            #     print("mouse_keyboard_connections() Error: "+ str(msg))
-            # continue
+        except Exception as msg:
             pass
 
-
+##
+# @brief This function is responsible for handling the camera frames
+##
 def camera_stream_connections():
+    """This function uses the 'OpenCV' library to decode and resize the camera frame. Then, this frame is scheduled on 
+    the virtual webcam device created using 'pyfakewebcam' library."""
     thread_run = True
     image_window_open=False
 
@@ -228,17 +224,11 @@ def camera_stream_connections():
             nparr = np.frombuffer(b, dtype=np.uint8)
             frame = cv2.imdecode(nparr, cv2.IMREAD_COLOR)
             img = cv2.resize(frame, (img_width, img_height))
-            #height, width = img.shape[:2]
             try:
-                # cv2.imshow('Main', img)
-                # image_window_open=True
-                # cv2.waitKey(5)
-
                 camera.schedule_frame(img)
 
             except Exception as msg:
                 print ("error: "+ str(msg))
-                #print("Frame height: "+ str(height) +"  width: "+ str(width))
             
             conn.close()
 
@@ -252,7 +242,12 @@ def camera_stream_connections():
                 pass
 
 
+##
+# @brief This function is responsible for listening to connections
+##
 def listening_connections():
+    """This function creates two threads corresponding to the two sockets, one for handling mouse and keyboard events 
+    and the other for handling camera frames received from the user's smartphone."""
     bind_sockets()
     mouse_thread=threading.Thread(target=mouse_keyboard_connections)
     mouse_thread.daemon=True

WirelessX-Python-Server/install.sh

+#!/bin/bash
+apt-get install python3-tk python3-dev
+pip3 install opencv-python
+pip3 install autopy
+pip3 install pynput
+pip3 install pyfakewebcam
+apt install ffmpeg
+apt install v4l2loopback-dkms v4l2loopback-utils
+modprobe -r v4l2loopback
+depmod -a