April 11, 2017 at 15:47SMS history: how to transmit text via voice channel
SMS is so familiar that many don’t even think about how messaging works in voice-oriented networks and don’t remember why we are limited by such a short message length. Let's remember where short messages came from and what they are.
This year, “short messages” will be 25 years old. It is believed that the first SMS was sent in the UK in 1992, and the commercial SMS service appeared in Finland in 1993.
It should be noted that before the advent of the GSM standard, SMS analogs did not exist, but by the current moment, i.e. after almost 25 years, the service has gained immense popularity, despite the fact that operators charge for outgoing and sometimes incoming messages.
Of course, the path to popularity for this service was rather thorny. For the first time, the idea of a service that transmits short messages between telephone sets appeared in 1984. The German engineer Friedhelm Hillebrand, who in 1985 headed the “non-voice” committee of GSM, already showed that 160 characters in Latin (which corresponds to 140 bytes) are enough to transmit most household messages. His statement was based on not-so-scientific experiments (hand-searching of random household questions and answers, as well as the observation that postcards and analyzed telex messages for business users almost always contain less than 150 characters), but they determined the nature of the service by dozens years.
Are 160 characters enough? Of course, Tolstoy’s proposals could not be accommodated in them, but before the on-screen keyboards appeared (on devices with touch screens) or at least T9, the users themselves did not seek to send longer text. Moreover, at the start, although SMS was laid down in the GSM standard, it was not uncommon to have support for dialing and sending short messages in some devices.
From a technical point of view, SMS has not undergone almost any changes (with the transition to GSM 03.40 - https://en.wikipedia.org/wiki/GSM_03.40 - perhaps only the set of supported characters in the message body has seriously changed). The commercial service developed in the networks of various operators approximately the same, adjusted for launch time, marketing and monetization approach.
Today, in addition to user messages on the network, a huge number of service messages from operators is transmitted (for example, delivery notifications), as well as service SMS from third-party service providers: banks, providers, Internet services. Moreover, they can be not only informative in nature (inform about the balance on the account, warn about the next payments under the contract, notify about the delivery of parcels), but also serve as one of the tools in two-factor authentication. In addition, short messages can be used to manage a number of services, for example, to post to Twitter.
An SMS message consists of text up to 140 bytes in length and a block of service information:
Subsequent fields may be present or absent, as well as interchanged, depending on the type of message. A detailed description of all fields can be found in the GSM 03.40 specification , here we will provide only the main ones, interesting from the point of view of parsing the message forwarding process:
Initially (in the GSM phase 2 document), each SMS message was a self-contained object, but according to subsequent editions (GSM phase 2+), a message transmitted in this way can include several messages - in this case, each of them contains a mark on the serial number (because of which the length of the "body" of each message is reduced).
Between individual users of a mobile network, a message is transmitted through an SMS center (SMSC or SMS-SC - Short Message Service - Service Center).
Together with the text of the message, additional information is transmitted from the mobile phone to the base station, including the address of the message center (in the early 2000s, this number had to be driven in during the initial setup of the purchased device). If a message travels between networks, SMS-center of the sender’s network is primarily responsible for its delivery.
SMSC not only delivers and forwards messages if necessary, but also stores them. If the recipient's phone is turned off at the time of sending the message, it remains in the SMSC until the device appears on the network. Typically, operators set the maximum storage period for such messages (technically, this setting may also be available to mobile network users). SMSC also sends message delivery reports, if specified in the user settings.
In the process of transmitting a message from one user to another, not only SMSC is involved. Briefly, the process is as follows:
Once on the recipient's phone, messages are displayed on the screen or stored on the SIM card / in the device’s memory, depending on the user settings and the properties of the message itself.
For historical reasons, telephony, including GSM, operates on the basis of the OKS-7 protocols (Common Signaling Channel - 7). In the terminology of this standard, SMS is transmitted over the signaling channel, and not over the traffic channel with voice data. This was conceived so that SMS could freely pass in parallel with the transmission of voice traffic (during a conversation). In addition, the signaling channel most of the time is “idle”, so transmitting short messages through it is a completely economical solution.
However, SMSC does not support OKS-7, therefore, for communication of the SMS center with other operator systems (HLR / VLR and MSC), additional gateways have to be used. To deliver messages to subscribers of other operators, special gateways are also used. The first gateway appeared in the world only in 1999, while we did not have a full-fledged message transfer between operators in the early 2000s (gateways appeared a bit later ).
With the advent of the need to connect a communication network (fixed or mobile) to
IP networks - at the turn of the century - the SIGTRAN specification was created, which allows transmitting OKS-7 streams over IP networks. So there were, including SMS gateways that work on the Internet. These gateways can not only "send" content from the Internet to SMS, but also allow the reverse transfer of data, for example, sending SMS messages from a phone to an email address (though the peak of the development of such services has already passed - with the advent of Internet access on phones , built-in email clients and applications for smartphones OS has become much easier to send emails directly).
SMS gateways can belong to the operator or exist independently, including as an addition to commercial corporate software for SMS mailings. Poor security of such gateways, as well as a very real opportunity to build a fraudulent gateway and lead to the spread of spam, hacking and other troubles associated with SMS. However, to the best of their ability and, to be sure, under pressure from the regulator, mobile operators are trying to deal with these problems.
Like any popular technology, during its existence, SMS has grown into a heap of legends and myths. Here are some interesting stories that managed to find confirmation:
Instead of a conclusion, I would like to note that although SMS has not been technologically developed for a long time, the peak of popularity in the world of the service has passed quite recently - in 2012-2014. Considering that the large-scale take-off of the technology occurred in the early 2000s, it stayed on the pedestal for an amazingly long time (from the point of view of a world so avid for novelties). What awaits an SMS message tomorrow - a drop in popularity or a stable future? Time will tell.
The development of SMS as a service
This year, “short messages” will be 25 years old. It is believed that the first SMS was sent in the UK in 1992, and the commercial SMS service appeared in Finland in 1993.
It should be noted that before the advent of the GSM standard, SMS analogs did not exist, but by the current moment, i.e. after almost 25 years, the service has gained immense popularity, despite the fact that operators charge for outgoing and sometimes incoming messages.
Of course, the path to popularity for this service was rather thorny. For the first time, the idea of a service that transmits short messages between telephone sets appeared in 1984. The German engineer Friedhelm Hillebrand, who in 1985 headed the “non-voice” committee of GSM, already showed that 160 characters in Latin (which corresponds to 140 bytes) are enough to transmit most household messages. His statement was based on not-so-scientific experiments (hand-searching of random household questions and answers, as well as the observation that postcards and analyzed telex messages for business users almost always contain less than 150 characters), but they determined the nature of the service by dozens years.
Are 160 characters enough? Of course, Tolstoy’s proposals could not be accommodated in them, but before the on-screen keyboards appeared (on devices with touch screens) or at least T9, the users themselves did not seek to send longer text. Moreover, at the start, although SMS was laid down in the GSM standard, it was not uncommon to have support for dialing and sending short messages in some devices.
From a technical point of view, SMS has not undergone almost any changes (with the transition to GSM 03.40 - https://en.wikipedia.org/wiki/GSM_03.40 - perhaps only the set of supported characters in the message body has seriously changed). The commercial service developed in the networks of various operators approximately the same, adjusted for launch time, marketing and monetization approach.
Today, in addition to user messages on the network, a huge number of service messages from operators is transmitted (for example, delivery notifications), as well as service SMS from third-party service providers: banks, providers, Internet services. Moreover, they can be not only informative in nature (inform about the balance on the account, warn about the next payments under the contract, notify about the delivery of parcels), but also serve as one of the tools in two-factor authentication. In addition, short messages can be used to manage a number of services, for example, to post to Twitter.
Message format
An SMS message consists of text up to 140 bytes in length and a block of service information:
- message type - 2 bits (SMS-DELIVER - message delivery to a mobile phone, SMS-DELIVER-REPORT - delivery confirmation, SMS-SUBMIT - sending a message from a mobile device, SMS-SUBMIT-REPORT - confirmation of sending, SMS-COMMAND - changing parameters or deleting a message stored in the SMS center, SMS-STATUS-REPORT - notification of the sender about the fate of his message).
Subsequent fields may be present or absent, as well as interchanged, depending on the type of message. A detailed description of all fields can be found in the GSM 03.40 specification , here we will provide only the main ones, interesting from the point of view of parsing the message forwarding process:
- destination address - 2-12 bytes ;
- date and time of sending (mark of the SMS center) - 7 bytes ;
- protocol identifier - 1 byte ;
- encoding scheme - 1 byte : encoding, message class (affects the way the device processes the message: is displayed without saving, is stored in the SIM or the device’s memory), a request for automatic deletion after reading, the status of flags corresponding to an unread fax or voice message, information compression of message content, language (for broadcast SMS);
- the length of the user data area is 1 byte ;
- message text - up to 140 bytes . The length of the message "in letters" varies, depending on the encoding used. These are 160 characters for 7-bit encoding (Latin + basic characters - in accordance with the GSM specification 03.38), 140 characters for 8-bit (Latin + diacritical characters) and 70 characters for UCS-2 (Russian operators work with this encoding, therefore, for Cyrillic characters, the message length is not more than 70 characters).
Initially (in the GSM phase 2 document), each SMS message was a self-contained object, but according to subsequent editions (GSM phase 2+), a message transmitted in this way can include several messages - in this case, each of them contains a mark on the serial number (because of which the length of the "body" of each message is reduced).
SMS messaging
Between individual users of a mobile network, a message is transmitted through an SMS center (SMSC or SMS-SC - Short Message Service - Service Center).
Together with the text of the message, additional information is transmitted from the mobile phone to the base station, including the address of the message center (in the early 2000s, this number had to be driven in during the initial setup of the purchased device). If a message travels between networks, SMS-center of the sender’s network is primarily responsible for its delivery.
SMSC not only delivers and forwards messages if necessary, but also stores them. If the recipient's phone is turned off at the time of sending the message, it remains in the SMSC until the device appears on the network. Typically, operators set the maximum storage period for such messages (technically, this setting may also be available to mobile network users). SMSC also sends message delivery reports, if specified in the user settings.
In the process of transmitting a message from one user to another, not only SMSC is involved. Briefly, the process is as follows:
- A request to send a message arrives at the mobile switching center (MSC),
- MSC checks whether the sender can work with messages on the GSM subscriber database (HLR / VLR - Home / Visitors Location Register - the general database and the database of a separate switch) and forwards the message along with the sender and receiver addresses to the SMSC;
- SMSC sends a confirmation of receipt, assigns the message a temporary status of “not delivered”,
- And makes an attempt to deliver the text to the addressee.
- Before delivery, the SMSC sender requests from the HLR a unique recipient identifier (IMSI - International Mobile Subscriber Identity), which is necessary for any operations with the subscriber within the network, and also checks for the prohibition of working with SMS. At the same time, the address of the MSC serving the subscriber is requested;
- After that, the location (LAI - Location Area Identity) and the temporary identifier (TMSI - temporary mobile subscriber identity) of the subscriber are requested from the MSC and a connection with the device is initiated to send the message directly;
- Upon completion of the transfer, the mobile phone sends a delivery confirmation (accordingly, the status of the message in the SMSC changes to "delivered"). And in case of unsuccessful delivery, the status “not delivered” is saved, while the reason is fixed in the HLR - the lack of memory or a subscriber in the network. Repeated delivery is initiated by the actions of the addressee (appearance on the network or clearing the device’s memory).
Once on the recipient's phone, messages are displayed on the screen or stored on the SIM card / in the device’s memory, depending on the user settings and the properties of the message itself.
Gateways, gateways ...
For historical reasons, telephony, including GSM, operates on the basis of the OKS-7 protocols (Common Signaling Channel - 7). In the terminology of this standard, SMS is transmitted over the signaling channel, and not over the traffic channel with voice data. This was conceived so that SMS could freely pass in parallel with the transmission of voice traffic (during a conversation). In addition, the signaling channel most of the time is “idle”, so transmitting short messages through it is a completely economical solution.
However, SMSC does not support OKS-7, therefore, for communication of the SMS center with other operator systems (HLR / VLR and MSC), additional gateways have to be used. To deliver messages to subscribers of other operators, special gateways are also used. The first gateway appeared in the world only in 1999, while we did not have a full-fledged message transfer between operators in the early 2000s (gateways appeared a bit later ).
With the advent of the need to connect a communication network (fixed or mobile) to
IP networks - at the turn of the century - the SIGTRAN specification was created, which allows transmitting OKS-7 streams over IP networks. So there were, including SMS gateways that work on the Internet. These gateways can not only "send" content from the Internet to SMS, but also allow the reverse transfer of data, for example, sending SMS messages from a phone to an email address (though the peak of the development of such services has already passed - with the advent of Internet access on phones , built-in email clients and applications for smartphones OS has become much easier to send emails directly).
SMS gateways can belong to the operator or exist independently, including as an addition to commercial corporate software for SMS mailings. Poor security of such gateways, as well as a very real opportunity to build a fraudulent gateway and lead to the spread of spam, hacking and other troubles associated with SMS. However, to the best of their ability and, to be sure, under pressure from the regulator, mobile operators are trying to deal with these problems.
Interesting facts about SMS
Like any popular technology, during its existence, SMS has grown into a heap of legends and myths. Here are some interesting stories that managed to find confirmation:
- In 2003, Malaysia recognized the legitimate SMS-divorce between spouses;
- In 2007, the number of SMS messages sent by the average American exceeded the number of calls he made;
- In 2008, SMS helped conduct a complex surgical operation for a patient from Congo: a volunteer surgeon received instructions from colleagues through text messages;
- Text abbreviations as part of SMS communication are becoming a new direction in the development of the language: in 2011, the popular English LOL and OMG were entered into the Oxford Dictionary;
- The German engineer F. Hillebrand, who proposed the idea of SMS, ultimately received only worldwide fame - the developers did not provide any license fees or patenting services.
Instead of a conclusion, I would like to note that although SMS has not been technologically developed for a long time, the peak of popularity in the world of the service has passed quite recently - in 2012-2014. Considering that the large-scale take-off of the technology occurred in the early 2000s, it stayed on the pedestal for an amazingly long time (from the point of view of a world so avid for novelties). What awaits an SMS message tomorrow - a drop in popularity or a stable future? Time will tell.
