URL Encoding Explained: Why Special Characters Need Encoding

What is URL Encoding?

URL encoding (percent-encoding) is a mechanism for representing characters that are either not allowed in a URL, or that have a special syntactic meaning and need to be treated as literal data rather than URL syntax. Each such character is replaced by a percent sign (%) followed by two hexadecimal digits representing the character's UTF-8 byte value.

For example:

• Space → %20
• & (ampersand) → %26
• = (equals) → %3D
• + (plus) → %2B
• / (forward slash) → %2F

RFC 3986: The URL Standard

The authoritative specification for URIs (Uniform Resource Identifiers, the superset of URLs) is RFC 3986, published in 2005 and still current. It defines the URI syntax, the allowed character sets, and when percent-encoding is required.

URLs have a defined structure: scheme://authority/path?query#fragment. Each component has specific rules about which characters are allowed. Characters that do not fit those rules must be percent-encoded.

Reserved vs Unreserved Characters

RFC 3986 divides characters into two main groups:

Unreserved Characters

These can always appear in a URL without encoding. They are: uppercase and lowercase letters (A–Z, a–z), digits (0–9), hyphen (-), underscore (_), period (.), and tilde (~).

Reserved Characters

These have special syntactic meaning in URLs and must be percent-encoded when used as literal data:

Character	URL Meaning	Percent-Encoded
:	Separates scheme from authority	%3A
/	Path separator	%2F
?	Starts query string	%3F
#	Starts fragment	%23
&	Separates query parameters	%26
=	Separates key from value in query	%3D
+	Space in form data	%2B
@	Separates userinfo from host	%40

How Percent-Encoding Works

Percent-encoding converts each character to its UTF-8 byte representation, then formats each byte as %XX where XX is the uppercase hexadecimal value of the byte.

For ASCII characters (code points 0–127), the UTF-8 byte is the same as the ASCII code. For example:

• Space (ASCII 32, hex 20) → %20
• ! (ASCII 33, hex 21) → %21
• @ (ASCII 64, hex 40) → %40

For multi-byte Unicode characters, each UTF-8 byte is encoded separately. The euro sign € (U+20AC) encodes as three UTF-8 bytes E2 82 AC, giving %E2%82%AC.

Query Strings and Parameters

The query string is the part of a URL after the ?. It contains key-value pairs separated by &:

https://example.com/search?q=hello+world&sort=date&order=desc

When a query parameter value contains reserved characters, those characters must be percent-encoded. Consider a search for "price > $10 & colour = blue":

https://example.com/search?q=price+%3E+%2410+%26+colour+%3D+blue

The >, $, &, and = in the search term are all encoded because they would otherwise be misinterpreted as URL syntax.

Form Data Encoding

When an HTML form is submitted with method="GET" or as application/x-www-form-urlencoded (the default for POST forms), browsers use a slightly different encoding from standard percent-encoding:

• Spaces are encoded as + rather than %20
• All other special characters are percent-encoded as normal

This is defined by the URL living standard and the original HTML form specification. The +-for-space convention applies only in application/x-www-form-urlencoded context — in regular URL paths, + is a literal plus sign.

This is a common source of bugs: a backend that decodes + as a space may work fine for form submissions but fail when processing URLs where + should be a literal plus.

Encoding Unicode Characters

Modern URLs can include non-ASCII characters (known as IRIs — Internationalized Resource Identifiers). However, for maximum compatibility, non-ASCII characters should be percent-encoded using their UTF-8 byte sequences.

Example: encoding the Japanese word 日本 (Nihon):

• 日 (U+65E5): UTF-8 bytes E6 97 A5 → %E6%97%A5
• 本 (U+672C): UTF-8 bytes E6 9C AC → %E6%9C%AC

Result: %E6%97%A5%E6%9C%AC

Modern browsers automatically encode non-ASCII characters in the address bar, so users can type international characters naturally. Server-side code must be prepared to decode UTF-8 percent-encoded sequences correctly.

JavaScript: encodeURI vs encodeURIComponent

JavaScript provides two functions for URL encoding, and choosing the wrong one is a common mistake:

encodeURIComponent() — encodes a single URL component value. It encodes all characters except: A–Z a–z 0–9 - _ . ! ~ * ' ( ). Use this for individual parameter values.

encodeURIComponent("price > $10 & colour = blue")
// Returns: "price%20%3E%20%2410%20%26%20colour%20%3D%20blue"

encodeURI() — encodes a complete URL. It does not encode characters that have syntactic meaning in URLs (: / ? # [ ] @ ! $ & ' ( ) * + , ; =). Use this only for full URL strings where the delimiters must stay intact.

encodeURI("https://example.com/search?q=hello world")
// Returns: "https://example.com/search?q=hello%20world"

The rule of thumb: use encodeURIComponent() for values, encodeURI() for complete URLs. In most real-world scenarios, you want encodeURIComponent().

Common Mistakes

Double-Encoding

Encoding an already-encoded string: %20 becomes %2520 (the % gets encoded). Always check whether input is already encoded before encoding again.

Not Encoding Plus Signs

If a value contains a literal +, encode it as %2B to prevent it being decoded as a space in form-encoding contexts.

Forgetting to Decode

Server-side code that reads URL parameters must decode them. Most frameworks do this automatically, but if you are reading raw request strings, call the appropriate decode function (decodeURIComponent in JavaScript, urllib.parse.unquote in Python, etc.).

Using encodeURI for Component Values

If you use encodeURI() on a parameter value, characters like &, =, and + will not be encoded, breaking the query string structure.