Building a Twitter Video Downloader Is Harder Than It Looks

URL normalization, media variants, and the backend boundaries that kept the integration maintainable I recently built a backend workflow that accepts public Twitter/X status links and turns them into predictable media records. The first prototype appeared to need only three steps: accept a URL, resolve the post, and return a video. Real inputs quickly made that design inadequate. Users paste both x.com and twitter.com links. Some URLs include /video/1, tracking parameters, whitespace, or the i/status form. A video post can expose several MP4 variants with different bitrates and dimensions. Image posts can contain multiple original images. Delivery URLs can expire, and a client can repeat the same request after a timeout even when the first attempt succeeded. The useful engineering problem was therefore not “download a video.” It was “turn an unstable public URL into a safe, idempotent job with a stable output contract.” Normalize before creating a job I validate the source URL before any network call. The service accepts HTTPS only, restricts the hostname to the supported Twitter/X hosts, strips fragments and known tracking parameters, and extracts the status identifier as a string. Keeping the identifier as a string is important. Status IDs can exceed the range in which every language and database driver represents integers safely. There is no benefit in doing arithmetic on them. I then create a canonical job key from the platform and status ID: twitter:2064933214720803143 This makes equivalent x.com and twitter.com links converge on the same record. It also stops repeated clicks from creating duplicate work. URL allowlisting is part of the security boundary, not only data cleanup. Any backend that fetches user-supplied URLs should treat server-side request forgery as a primary threat. The OWASP SSRF prevention guidance is a useful baseline: use an allowlist, reject internal destinations, and do not build a general-purpose proxy around arbitrary input. Keep parsing and credentials on the server The browser sends the public status URL to my application. A server route or worker performs the parsing request. Tokens never appear in client-side JavaScript, browser storage, or query strings. The request shape is deliberately small: curl --request POST \ --url https://api.easydown.org/api/v1/platforms/twitter/parse \ --header "Authorization: Bearer $EASYDOWN_API_TOKEN" \ --header "Content-Type: application/json" \ --data '{"url":"https://x.com/i/status/STATUS_ID"}' While building this workflow in EasyDown, I documented the supported URL shapes, endpoint, normalized response, and platform fields in the Twitter video downloader API documentation. The scope remains explicit: public status links and content the user is authorized to process. Private accounts, account-wide collection, and arbitrary profile crawling are not part of the importer. I store the response in two layers. The first layer is a normalized media record used by the rest of the application. It contains a platform name, title, thumbnail, duration, images, videos, and audio. A generic queue or UI can consume this layer without knowing how Twitter/X represents attachments. The second layer contains sanitized platform-specific fields. Depending on the public post, those fields can include the string ID, expanded post text, public author data, attached media objects, public engagement counters, and a platform data version. Versioning this second layer matters. Platform data is useful, but it changes more often than the normalized contract. The application should not scatter provider-specific assumptions across unrelated code. A common shortcut is to return the first MP4 URL. That makes the API appear simple while hiding the choice that the application actually needs to make. Twitter/X video posts can expose multiple variants. I retain each MP4 candidate with its bitrate, dimensions when available, MIME type, and audio flag. The consuming service can then choose the best variant for its own constraints. For an interactive download, the highest practical bitrate may be the right default. For a mobile preview, a smaller rendition may reduce latency. For archiving authorized content, the system may preserve the original candidate list and select a format later. Image posts follow the same principle: keep every post image in order. A thumbnail is presentation metadata and should not silently replace downloadable post media. Make retries safe at two levels The importer has two kinds of retry. The first retry is the client repeating a request because it did not receive a response. The second is the worker retrying a temporary upstream failure. Both can create duplicates unless the operation is idempotent. I derive an idempotency key from the normalized status ID and operation type. A database uniqueness constraint ensures that two concurrent requests converge on the same job. The worker records a state such as queued, parsing, ready, or failed, and a retry continues from the existing record. The distinction between safe repetition and blind repetition is easy to miss. The HackerNoon article Idempotency, APIs, and Retries — Oh My! gives a useful explanation of why retries for non-idempotent operations require cooperation from the server. The HTTP semantics specification also defines idempotent methods, but an application-level POST can still be made retry-safe with a stable operation key. I retry only temporary failures: timeouts, rate limits, and transient upstream errors. Authentication, invalid URL, unsupported content, and insufficient balance errors remain terminal until the request or configuration changes. Exponential backoff and a hard attempt limit prevent a failing platform from turning into an internal traffic storm. Resolved media URLs are not durable identifiers. I enqueue downstream work immediately after a successful parse. If the product needs permanent storage, a worker copies authorized media into controlled object storage and keeps the normalized source URL as provenance. The browser never receives a general proxy endpoint. The backend fetcher enforces host allowlists, content-length limits, timeouts, and allowed MIME types. It also rejects redirects that escape the approved host set. Log enough to operate the system For each parse job I record the normalized status ID, platform, latency, cache outcome, selected variant, attempt count, and error category. I do not log bearer tokens or complete signed delivery URLs. Those fields answer the operational questions that matter: Is the platform failing or only one URL shape? Are rate limits increasing? Are retries recovering requests or multiplying load? Which bitrate is the application selecting? Are users submitting unsupported links? The boundary is the product The most valuable part of this integration is not the parser by itself. It is the boundary around it: Normalize equivalent public URLs. Keep large status IDs as strings. Restrict fetches to approved hosts. Keep tokens and parsing calls on the server. Preserve all useful media variants. Separate normalized media from versioned platform data. Make client and worker retries idempotent. Treat delivery URLs as temporary. Once those decisions are explicit, adding another platform becomes a contract-mapping problem instead of a rewrite. The application can keep one stable import pipeline while platform-specific code remains contained. Disclosure: I work on EasyDown, the API used in the server request example above.