Commit 17d9251a authored by rajneesh's avatar rajneesh

Python Server side comments added and readme updated

parent f1f916ad
# WIRELESS X
## Installation Setup
1. Make sure that you are in the Wireless-X source directory
2. Grant the permission to execute install.sh installation script using the following command:
$ sudo chmod a+x install.sh
3. Execute the install.sh script to install the necessary dependencies using the following command:
$ sudo ./install.sh
## Running the application (Strictly follow the below order to run it successfully):
1. Run the Wireless-X server using the following command:
$ python3 Wireless-X_server.py
2. Enter your linux system password (the same password you enter while executing a command as "sudo"). This is required in order to set up the virtual webcam device.
3. Application Installation and Setup on Android Smartphone:
a. Install the Wireless-X apk on Android smartphone and give required permissions.
b. Now enter the IP address of the Laptop/ Desktop on which the server is hosted. You can type the below command in terminal to get host IP:
$ hostname -I | awk '{print $1}'
c. Click on Test Button to test the connection of smartphone with the server. If failed, Recheck if you have entered correct IP address of Laptop/ Desktop (on which the server is running).
d. After successfull connection, you would be able to control mouse, keyboard of laptop and use smartphones camera as webcam for the laptop/Desktop.
e. Now you would be able to use this virtual webcam device on chrome for video conferincing. (Tested on chrome for MS Teams and Google Meet).
f. (Optional) Inorder to test if camera frames are received to the Laptop/ Desktop, use the below command while (Note: camera option should be turned on in the
Wireless-X apk on Android):
$ ffplay /dev/video20
#### *Extra (Inorder to remove v42loopback devices, use below command):
$ sudo modprobe -r v4l2loopback
## Steps for Debugging (If python code doesn't run after above commands):
1. Check if your virtual device is created
$ ls /dev | grep -P '^video\d+$'
OR
$ v4l2-ctl --list-devices # TO List the virtual devices in detail
Output should look somewhat like this:
Wireless-X Camera (platform:v4l2loopback-000):
/dev/video20
Webcam C170: Webcam C170 (usb-0000:00:1a.0-1.2):
/dev/video0
/dev/video1
2. Inorder to test if virtual device is working:
Copy the sample code from https://github.com/jremmons/pyfakewebcam page and save it as python file and run it.
$ python3 demo.py
If everything worked correctly, no error should be displayed and terminal should be blank.
Now, Open another terminal and test if virtual device output is being display by entering below command:
$ ffplay /dev/video20
Note: ffplay
## References
OpenCV Reference:
https://cmsdk.com/java/android-opencv-tcp-video-streaming.html
v4l2loopback References:
https://github.com/jremmons/pyfakewebcam
https://github.com/jremmons/pyfakewebcam/issues/5
https://github.com/umlaeute/v4l2loopback#DKMS
Android Reference:
https://stackoverflow.com/questions/23024831/android-shared-preferences-for-creating-one-time-activity-example
\ No newline at end of file
##
# @file
# This includes the code for the server-side of Wireless-X
#
# The server running on laptop or PC is responsible for receiving the actions performed by user on the
# Wireless-X android app as well as receiving the camera frames of the user's smartphone (if the user has turned)
# on the camera). Such actions are transmitted to the server in encoded form, the server decodes the received
# message and instructs the laptop or PC to perform the action described in that message.
##
import socket import socket
import time import time
import threading import threading
...@@ -6,36 +16,72 @@ import cv2 ...@@ -6,36 +16,72 @@ import cv2
import numpy as np import numpy as np
import binascii import binascii
import pyfakewebcam import pyfakewebcam
import subprocess
from pynput.keyboard import Key, Controller as KeyboardController from pynput.keyboard import Key, Controller as KeyboardController
from pynput.mouse import Button, Controller as MouseController from pynput.mouse import Button, Controller as MouseController
print('---------Wireless-X Server------') ## Creates a virtual camera on the laptop/PC
virtualCamera = subprocess.run(["sudo", "modprobe", "v4l2loopback", "devices=1", "video_nr=20", "card_label='Wireless-X Camera'", "exclusive_caps=1"])
print("Virtual camera status:", virtualCamera.returncode)
print('----------Wireless-X Server----------')
print('Press Ctrl+C to terminate the server')
## @var width
# Stores the width of the screen
## @var height
# Stores the height of the screen
width, height = autopy.screen.size() width, height = autopy.screen.size()
print("width: "+str(width)+" height: "+str(height)) print("width: "+str(width)+" height: "+str(height))
## @var curr_x
# Stores the x-coordinate of the current mouse location
## @var curr_y
# Stores the y-coordinate of the current mouse location
curr_x, curr_y = autopy.mouse.location() curr_x, curr_y = autopy.mouse.location()
## Stores the mid of x-coordinate of current mouse location
remote_x = curr_x/2 remote_x = curr_x/2
## Stores the mid of y-coordinate of current mouse location
remote_y = curr_y/2 remote_y = curr_y/2
## Socket used for receiving keyboard and mouse related actions
s='' s=''
## Socket used for receiving the camera frames of user's smartphone
cameraSocket='' cameraSocket=''
## Width of the camera frame
img_width = 720 img_width = 720
## Height of the camera frame
img_height = 480 img_height = 480
## Virtual webcam device
camera = pyfakewebcam.FakeWebcam('/dev/video20', img_width, img_height) camera = pyfakewebcam.FakeWebcam('/dev/video20', img_width, img_height)
## The camera and keyboard-mouse sockets receive user requests until this variable is set to 'True'
thread_run = True thread_run = True
## Initializing the KeyboardController object
keyboard = KeyboardController() keyboard = KeyboardController()
mouse = MouseController()
mouse_speed =2 ## Initializing the MouseController object
mouse = MouseController()
## Speed of mouse movement
mouse_speed = 2
screenshot_count=0 ## Screenshot counter to keep track of screenshots
screenshot_count = 0
# binding the socket ##
# @brief This function establishes two sockets for receiving camera frames as well as mouse and keyboard actions
##
def bind_sockets(): def bind_sockets():
"""This function creates a camera socket which is responsible for receiving the camera frames, and it also
creates another socket which is responsible for receiving the keyboard and mouse frames."""
try: try:
global s, cameraSocket global s, cameraSocket
cameraSocket = socket.socket(socket.AF_INET, socket.SOCK_STREAM) cameraSocket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
...@@ -56,51 +102,24 @@ def bind_sockets(): ...@@ -56,51 +102,24 @@ def bind_sockets():
print("Keyboard and Mouse Listening port: "+ str(key_port)) print("Keyboard and Mouse Listening port: "+ str(key_port))
s.listen(10) s.listen(10)
except socket.error as msg: except socket.error as msg:
print("Socket Binding error: "+str(msg)) print("Socket Binding error: "+str(msg))
time.sleep(5) # Time to sleep before reattempting the bind connection time.sleep(5) # Time to sleep before reattempting the bind connection
bind_sockets() bind_sockets()
## Maps the keys in keyboard layout to the actual keyboard keys
'''
Testing Zone
jsjsjjjsu
rajeesh n rajneesh katkam
~12838
qwertyuiop[]
ASDFasdfghjkl;'
zxcvbnm, ./
zxcvbnzxcrvb<<<<<<<>>>>>>><<<<<~~~~~u@(@@*@wjwjsjiwi@u<><<<<<<<<<<<<<<<<>>>>>>>>>>>>>>)
Te84848 QWEqweUTREYYUWYWqtwywywuiiwywyRTWWYYUUUYETUIOIwwertyuiop[][
special_key_list = ["F1", "F2", "F3", "F4", "F5", "F6", "F7", "F8", "F9", "F10", "F11", "F12", "ALT", "BACKSPACE", "CAPS_LOCK", "CONTROL", "DELETE", "DOWN_ARROW", "END", "ESCAPE", "HOME", "LEFT_ARROW", "META", "PAGE_DOWN", "PAGE_UP", "RETURN", "RIGHT_ARROW", "SHIFT", "SPACE", "UP_ARROW"]
special_key_android_list = ["F1", "F2", "F3", "F4", "F5", "F6", "F7", "F8", "F9", "F10", "F11", "F12", "Alt", "Backspace", "Caps\nLock", "Ctrl", "Delete", "↓", "End", "Esc", "Home", "←", "META", "Page Down", "Page Up", "Enter", "→", "Shift", "Space", "↑"]
special_key_android_dictionary = {
"Alt":"ALT", "Caps\nLock":"CAPS_LOCK", "Ctrl":"CONTROL",
"META":"META",
"Enter":"RETURN", "Shift":"SHIFT", "Space":"SPACE", "Tab":"Tab"}
'''
special_key_android_dictionary = {"F1": "F1", "F2":"F2", "F3":"F3", "F4":"F4", "F5":"F5", "F6":"F6", "F7":"F7", "F8":"F8", "F9":"F9", "F10":"F10", "F11":"F11", "F12":"F12", "Alt":"ALT", "Backspace":"BACKSPACE", "Caps\nLock":"CAPS_LOCK", "Ctrl":"CONTROL", "Delete":"DELETE", "↓":"DOWN_ARROW", "End":"END", "Esc":"ESCAPE", "Home":"HOME", "←":"LEFT_ARROW", "META":"META", "Page Down":"PAGE_DOWN", "Page Up":"PAGE_UP", "Enter":"RETURN", "→":"RIGHT_ARROW", "Shift":"SHIFT", "Space":"SPACE", "↑":"UP_ARROW", "Tab":"Tab"} special_key_android_dictionary = {"F1": "F1", "F2":"F2", "F3":"F3", "F4":"F4", "F5":"F5", "F6":"F6", "F7":"F7", "F8":"F8", "F9":"F9", "F10":"F10", "F11":"F11", "F12":"F12", "Alt":"ALT", "Backspace":"BACKSPACE", "Caps\nLock":"CAPS_LOCK", "Ctrl":"CONTROL", "Delete":"DELETE", "↓":"DOWN_ARROW", "End":"END", "Esc":"ESCAPE", "Home":"HOME", "←":"LEFT_ARROW", "META":"META", "Page Down":"PAGE_DOWN", "Page Up":"PAGE_UP", "Enter":"RETURN", "→":"RIGHT_ARROW", "Shift":"SHIFT", "Space":"SPACE", "↑":"UP_ARROW", "Tab":"Tab"}
##
# @brief This function decodes the received mouse and keyboard actions and acts accordingly
##
def mouse_keyboard_connections(): def mouse_keyboard_connections():
"""This function checks if the message received corresponds to a mouse action (left-click, scroll, etc.) or a keyboard action
(such as keypress). It then instructs the laptop to perform these actions, using the 'autopy' and 'pynput' libraries. Some special
characters in keyboard are not supported by 'autopy' library, so the actions corresponding to these special characters are performed
by the 'pynput' library, other key actions are handled by the 'autopy' library. In case of mouse, the 'autopy' library was not that
efficient, so we used the 'pynput' library."""
global thread_run, curr_x, curr_y, remote_x, remote_y global thread_run, curr_x, curr_y, remote_x, remote_y
global screenshot_count global screenshot_count
thread_run=True thread_run=True
...@@ -120,31 +139,14 @@ def mouse_keyboard_connections(): ...@@ -120,31 +139,14 @@ def mouse_keyboard_connections():
xy=peer_response.split(',') xy=peer_response.split(',')
if len(xy)==2: if len(xy)==2:
mouse.position = (float(xy[0])*width, float(xy[1])*height) mouse.position = (float(xy[0])*width, float(xy[1])*height)
#mouse.position = (float(xy[0])*mouse_speed, float(xy[1])*mouse_speed)
#mouse.move((float(xy[0])-0.5)*10, (float(xy[0])-0.5)*10)
#print(xy[1]+","+xy[0], end="\t")
#print(str(width), str(height), end="\t")
#remote_x= float(max(min(float(xy[0]),1),0)*width) # + prev_x
#remote_y= float(max(0,min(float(xy[1]),1),0)*height) # + prev_y
#autopy.mouse.move(remote_x, remote_y)
#mouse.position = (remote_x, remote_y)
elif "!#Keyboard#!" in peer_response: elif "!#Keyboard#!" in peer_response:
peer_response=peer_response.split("!#Keyboard#!")[1] peer_response=peer_response.split("!#Keyboard#!")[1]
#print("Key:"+ peer_response)
#autopy.key.tap(autopy.key.Code.TAB, [autopy.key.Modifier.META])
#autopy.key.tap(autopy.key.Code.RETURN)
if peer_response in special_key_android_dictionary.keys(): if peer_response in special_key_android_dictionary.keys():
print("Special: "+ special_key_android_dictionary[peer_response]) print("Special: "+ special_key_android_dictionary[peer_response])
if peer_response == "Backspace": if peer_response == "Backspace":
#pyautogui.press('backspace')
keyboard.press(Key.backspace) keyboard.press(Key.backspace)
keyboard.release(Key.backspace) keyboard.release(Key.backspace)
...@@ -160,7 +162,6 @@ def mouse_keyboard_connections(): ...@@ -160,7 +162,6 @@ def mouse_keyboard_connections():
elif peer_response == "Print\nScreen": elif peer_response == "Print\nScreen":
print("Entered Print Screen: "+ str(screenshot_count)) print("Entered Print Screen: "+ str(screenshot_count))
#autopy.bitmap.capture_screen().save(str("screenshot.png"))
autopy.bitmap.capture_screen().save(str("screenshot_"+str(screenshot_count)+".png")) autopy.bitmap.capture_screen().save(str("screenshot_"+str(screenshot_count)+".png"))
screenshot_count+=1 screenshot_count+=1
...@@ -190,25 +191,20 @@ def mouse_keyboard_connections(): ...@@ -190,25 +191,20 @@ def mouse_keyboard_connections():
else: else:
mouse.scroll(0, -2) # Scroll Down mouse.scroll(0, -2) # Scroll Down
elif "!#Test#!" in peer_response: elif "!#Test#!" in peer_response:
#print(peer_response)p conn.send(bytes("Success "+str(int(width/mouse_speed))+" "+str(int(height/mouse_speed)),"utf-8"))
conn.send(bytes("Success "+str(int(width))+" "+str(int(height)),"utf-8"))
# else:
# print(peer_response)
conn.close() conn.close()
except Exception as msg : except Exception as msg:
# if "timed out" not in str(msg):
# print("mouse_keyboard_connections() Error: "+ str(msg))
# continue
pass pass
##
# @brief This function is responsible for handling the camera frames
##
def camera_stream_connections(): def camera_stream_connections():
"""This function uses the 'OpenCV' library to decode and resize the camera frame. Then, this frame is scheduled on
the virtual webcam device created using 'pyfakewebcam' library."""
thread_run = True thread_run = True
image_window_open=False image_window_open=False
...@@ -228,17 +224,11 @@ def camera_stream_connections(): ...@@ -228,17 +224,11 @@ def camera_stream_connections():
nparr = np.frombuffer(b, dtype=np.uint8) nparr = np.frombuffer(b, dtype=np.uint8)
frame = cv2.imdecode(nparr, cv2.IMREAD_COLOR) frame = cv2.imdecode(nparr, cv2.IMREAD_COLOR)
img = cv2.resize(frame, (img_width, img_height)) img = cv2.resize(frame, (img_width, img_height))
#height, width = img.shape[:2]
try: try:
# cv2.imshow('Main', img)
# image_window_open=True
# cv2.waitKey(5)
camera.schedule_frame(img) camera.schedule_frame(img)
except Exception as msg: except Exception as msg:
print ("error: "+ str(msg)) print ("error: "+ str(msg))
#print("Frame height: "+ str(height) +" width: "+ str(width))
conn.close() conn.close()
...@@ -252,7 +242,12 @@ def camera_stream_connections(): ...@@ -252,7 +242,12 @@ def camera_stream_connections():
pass pass
##
# @brief This function is responsible for listening to connections
##
def listening_connections(): def listening_connections():
"""This function creates two threads corresponding to the two sockets, one for handling mouse and keyboard events
and the other for handling camera frames received from the user's smartphone."""
bind_sockets() bind_sockets()
mouse_thread=threading.Thread(target=mouse_keyboard_connections) mouse_thread=threading.Thread(target=mouse_keyboard_connections)
mouse_thread.daemon=True mouse_thread.daemon=True
......
#!/bin/bash
apt-get install python3-tk python3-dev
pip3 install opencv-python
pip3 install autopy
pip3 install pynput
pip3 install pyfakewebcam
apt install ffmpeg
apt install v4l2loopback-dkms v4l2loopback-utils
modprobe -r v4l2loopback
depmod -a
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment