May 30, 2011 at 15:38Touchpad and keyboard from a handy device on Windows Mobile
Do you have a communicator with WM and the need to control a PC with it?
Maybe you want to make some kind of tricky presentation that requires a mouse in your pocket?
Ok, you need a cut (there are pictures).
If you are only interested in the final product, but not in technical details, I recommend that you do not hesitate to go to the FAQ section.
Task: turn a WM-communicator into a full touchpad and keyboard, while gaining the ability to control any win-system with its help.
In general, the managed system may not be win, but more on that below.
We will use UDP / IP over Wi-Fi as a transport, or (for connoisseurs) Bluetooth.
What is required for this:
Written for .NET Compact Framework 3.5 in C # , tested on HTC HD2 with WM 6.5 .
The client application does not use any vendor-specific APIs, so it should work on all devices with WM 5 and 6.
The principle of capturing keyboard input and touchscreen is extremely simple:
A little about the terms in which the principle of emulation of the touchpad is expressed - they are used in the following description:
ConfigApply and ConfigStore are not directly relevant; all work with MouseHandler comes down to processing its events, whose purpose is generally clear from the names.
The device is trivial:
Events:
A class whose task is to generate control messages for sending to the server:
We receive the requested action through a method of the type KeyboardCommand (), MouseMove (), etc; we compose the message, send it to the socket through the OnPacketSend event.
The user is presented with a unique opportunity to edit their configs in a rudimentary XML format.
How the application implements work with them, see for yourself if anyone is interested. I'd rather not talk about that.
Screen: Assignment of the main fields of configs, not obvious from the name:
When drawing the interface, I was inspired by the popular picture of Kazimir Malevich: Since the GUI is considered to be a self-documenting user interface, I still left the button signatures (screen on the left) that disappear after the first touch of the touchpad (screen on the right). Yes, I have not said yet: this black field on the screen is the touchpad; the purpose of the buttons should be obvious from their name. Using a keyboard should also not cause issues. The portrait orientation of the window is used to save the screen area for the touchpad; while the use of the touchpad itself is assumed in landscape (but this is customizable).
The first attempt to create a more or less cross-platform server using python and SendKeys failed, as it turned out that SendKeys does not know Unicode. How to work from a python with a mouse, I did not even look.
The second attempt (successful) was to use WinAPI from a console application in C, where the work of writing commands to the mouse and keyboard stream is performed using the SendInput function .
A simple console application listening on a UDP socket and periodically calling SendInput () is not worth considering. I only note that it practically does not consume system resources (especially when the client is disconnected), so that it can be kept running constantly.
As for portability, it’s not difficult to write a platform-specific application for any other platform (the audience would need it), because it uses an easy-to-implement implementation to deliver commands to a managed system
Data is transmitted only from the client to the server, no acknowledgment or reverse commands are provided.
Any command is sent in a frame of a fixed size, consisting of the following fields:
Q: Ok, I want to use Device2Keyboard. What do I need to do for this?
A: Download the client and server applications using the links at the end of the post, and launch them. The default settings assume that the communicator and the server must be on the same subnet, since the default server address is 255.255.255.255:12358 (broadcast IP address) on the client. Everything will work out of the box, no additional configuration is required.
Q: I do not want to use wireless interfaces. How to be
A: Connect the device to a managed PC with ActiveSync installed, leaving the default settings for the client. Everything will work.
Q:I fixed the click while executing Drag'n'Drop, but the connection to the server was interrupted and the click remained clamped. What to do?
A: Wait 3 seconds. If the server does not receive commands from the client to set the state of mouse clicks during this time, it will release all the buttons that are held.
Q: If you move the mouse cursor for a long time, my network will fall due to heavy traffic, right?
A: The default frequency for updating the cursor position is 10ms (100 times per second); packet size (3 + 1 + 1 + 4) == 9 bytes, + UDP / IP frame overhead 28 bytes, we get 37 bytes in 1 message, or 3.6 kbps. If the mouse does not move, its position is not updated. The answer is negative.
Application for WM and desktop, installation does not require: dl.dropbox.com/u/3815390/dev2kbd/bin/bin.7z
Sources: dl.dropbox.com/u/3815390/dev2kbd/src/src.7z
Application and its sources distributed under the WTFPL license .
Photo from the title, source: migdal-or.livejournal.com/55703.html
Maybe you want to make some kind of tricky presentation that requires a mouse in your pocket?
Ok, you need a cut (there are pictures).
If you are only interested in the final product, but not in technical details, I recommend that you do not hesitate to go to the FAQ section.
Concept
Task: turn a WM-communicator into a full touchpad and keyboard, while gaining the ability to control any win-system with its help.
In general, the managed system may not be win, but more on that below.
We will use UDP / IP over Wi-Fi as a transport, or (for connoisseurs) Bluetooth.
What is required for this:
- capture input from the default keyboard of the device;
- using the touchscreen of a communicator emulate:
- regular touchpad, with scroll and clicks;
- mouse wheel;
- mouse buttons (with the ability to move the cursor with a clamped click for drag'n'drop);
- an application on a managed system (server) capable of writing data received from the device to the keyboard and mouse stream;
- A protocol for exchanging a client application with a server on a managed system.
Application for WM
Written for .NET Compact Framework 3.5 in C # , tested on HTC HD2 with WM 6.5 .
The client application does not use any vendor-specific APIs, so it should work on all devices with WM 5 and 6.
The principle of capturing keyboard input and touchscreen is extremely simple:
- We create some kind of control, and stretch it to fit the window (Dock = Fill);
- We put a focus on him;
- Subscribe to KeyDown, KeyUp, KeyPress events to intercept keystrokes;
- MouseDown, MouseUp, MouseMove to emulate the touchpad.
Touchpad
A little about the terms in which the principle of emulation of the touchpad is expressed - they are used in the following description:
- Static Move - if the user dragged his finger to the border of the touchpad and left it there, the controlled cursor will slowly creep in the direction of this border. Convenient for dragging the cursor with a clamped click;
- Scroll - the user ran a finger over a special region at the edge of the touchpad, which is interpreted as scrolling the wheel;
- Scroll Pad - the above-mentioned "special region" at the edge of the touchpad. If the finger starts moving on it, then scrolling is turned on, otherwise it works like a normal touchpad.
- Inertial Scrolling - the user activated scrolling, and removed his finger without stopping his movement. Scrolling will be performed further at the same speed without user intervention, until he is able to touch the touchpad again.
- Click and Drag - clicked, removed your finger, and immediately clicked again - now you can steer the cursor with a click.
ConfigApply and ConfigStore are not directly relevant; all work with MouseHandler comes down to processing its events, whose purpose is generally clear from the names.
Keyboard
The device is trivial:
Events:
- OnKeyCommand - received a control character, one of these . In general, they are standardized.
- OnKeyUnicode - received a printable character. K.O. notices that in Unicode.
Client
A class whose task is to generate control messages for sending to the server:
We receive the requested action through a method of the type KeyboardCommand (), MouseMove (), etc; we compose the message, send it to the socket through the OnPacketSend event.
Configs
The user is presented with a unique opportunity to edit their configs in a rudimentary XML format.
How the application implements work with them, see for yourself if anyone is interested. I'd rather not talk about that.
Screen: Assignment of the main fields of configs, not obvious from the name:
- buttonRightClickHold - show the right-click lock button. Those. when pressed, the right click sticks until you click again. By default, the button is not shown.
- buttonSettings - show the button for calling the laconic config editor, whose screenshot is above. By default, this button is for some reason turned on.
- clickAndDragDelay - the time during which you need to re-click on the touchpad to get Click and Drag (description above).
- locationUpdateInterval - interval to update the cursor position, in milliseconds.
- staticSpeed - speed of Static Move, pixels per packet.
- staticBoundsRelative - borders of the field on which Static Move is activated; are set relative to the overall window size.
- scrollInMousePixels - sensitivity of the Scroll Pad; the lower the number, the higher the sensitivity.
Interface
When drawing the interface, I was inspired by the popular picture of Kazimir Malevich: Since the GUI is considered to be a self-documenting user interface, I still left the button signatures (screen on the left) that disappear after the first touch of the touchpad (screen on the right). Yes, I have not said yet: this black field on the screen is the touchpad; the purpose of the buttons should be obvious from their name. Using a keyboard should also not cause issues. The portrait orientation of the window is used to save the screen area for the touchpad; while the use of the touchpad itself is assumed in landscape (but this is customizable).
Server
The first attempt to create a more or less cross-platform server using python and SendKeys failed, as it turned out that SendKeys does not know Unicode. How to work from a python with a mouse, I did not even look.
The second attempt (successful) was to use WinAPI from a console application in C, where the work of writing commands to the mouse and keyboard stream is performed using the SendInput function .
A simple console application listening on a UDP socket and periodically calling SendInput () is not worth considering. I only note that it practically does not consume system resources (especially when the client is disconnected), so that it can be kept running constantly.
As for portability, it’s not difficult to write a platform-specific application for any other platform (the audience would need it), because it uses an easy-to-implement implementation to deliver commands to a managed system
Protocol
Data is transmitted only from the client to the server, no acknowledgment or reverse commands are provided.
Any command is sent in a frame of a fixed size, consisting of the following fields:
- header - 3 bytes, string "d2k"
- serial number of the packet (to avoid packet retries when broadcasting, which is used by default) - 1 byte
- type of command (control character, unicode character, move mouse, etc) - 1 byte
- command data (cursor coordinates, character code, etc) - 4 bytes, little endian
FAQ
Q: Ok, I want to use Device2Keyboard. What do I need to do for this?
A: Download the client and server applications using the links at the end of the post, and launch them. The default settings assume that the communicator and the server must be on the same subnet, since the default server address is 255.255.255.255:12358 (broadcast IP address) on the client. Everything will work out of the box, no additional configuration is required.
Q: I do not want to use wireless interfaces. How to be
A: Connect the device to a managed PC with ActiveSync installed, leaving the default settings for the client. Everything will work.
Q:I fixed the click while executing Drag'n'Drop, but the connection to the server was interrupted and the click remained clamped. What to do?
A: Wait 3 seconds. If the server does not receive commands from the client to set the state of mouse clicks during this time, it will release all the buttons that are held.
Q: If you move the mouse cursor for a long time, my network will fall due to heavy traffic, right?
A: The default frequency for updating the cursor position is 10ms (100 times per second); packet size (3 + 1 + 1 + 4) == 9 bytes, + UDP / IP frame overhead 28 bytes, we get 37 bytes in 1 message, or 3.6 kbps. If the mouse does not move, its position is not updated. The answer is negative.
Conclusion
Application for WM and desktop, installation does not require: dl.dropbox.com/u/3815390/dev2kbd/bin/bin.7z
Sources: dl.dropbox.com/u/3815390/dev2kbd/src/src.7z
Application and its sources distributed under the WTFPL license .
Photo from the title, source: migdal-or.livejournal.com/55703.html