Migrating a Production Django App from Elasticsearch to OpenSearch

Migrating a Production Django App from Elasticsearch to OpenSearch

Key Takeaways Legacy search clients like elasticsearch==7.10.1 hard-cap underlying transport libraries (urllib3=1.26.19), requiring an explicit application-level pin of urllib3>=2.5.0. When third-party packages like django-elasticsearch-dsl-drf are abandoned, vendoring a lean (~190 LOC) project-local filter and pagination layer prevents breaking client-facing REST API response contracts. Verifying a search client migration requires separating metadata ping traffic (size=0) from document retrieval queries and verifying vulnerability remediation with automated security auditing tools. When a security advisory hits a core library like urllib3, upgrading is usually a routine dependency bump. However, when legacy search clients hard-cap underlying transport libraries, security remediations become full-scale client migrations. This is the story of that digging: how a security patch that looked like a one-line version bump turned into a full client-library migration, what part of that migration was genuinely mechanical, and what part wasn't. If you're running an older elasticsearch Python client against an AWS OpenSearch domain, there's a decent chance you're sitting on the same trap. The Anatomy of a Transitive CVE Trap In modern Python backend architectures, transitive dependencies can quietly turn into security bottlenecks. Our production Django application relied on elasticsearch==7.10.1 to interact with event and audit-log indices hosted on AWS OpenSearch 1.3. A transitive dependency is a package your code never imports directly, it gets pulled in because something you depend on needs it. That distance is exactly why a cap like this goes unnoticed for years: nobody on the team ever wrote import urllib3, so nobody was watching its version. When CVE-2025-50181 (an open-redirect / SSRF vulnerability in urllib3, affecting everything before 2.5.0, details here) was disclosed, security compliance required updating urllib3 to version 2.5.0 or higher. However, running a dependency audit revealed a hard blocker: # pyproject.toml (legacy state) elasticsearch = "7.10.1" # Pinned: requires urllib3 >= 1.21.1, =1.21.1,=1.21.1,=1.26.18 (cap dropped, undeclared) Unofficially, briefly 2.5.0 >=1.26.18,=1.26.18 Yes, first real support 2.8.0 (latest 2.x at the time) !=2.2.0,!=2.2.1,=1.26.19 Yes, plus more fixes 3.0.0 and later same floor Yes, but enforces keyword-only arguments, a bigger diff We landed on 2.8.0: same migration effort as 2.6.0, but with roughly six more months of bug and security fixes, while still avoiding the breaking change in 3.0 (which makes arguments like body= keyword-only, not removed, just no longer allowed positionally, which matters if any of your call sites pass it positionally). In dependency terms, >=1.26.19 is a floor: it says "at least this version," not "exactly this version." Package maintainers almost always publish floors rather than pins, because a pin would make their library incompatible with anything else in your project that needs a different exact version. That's exactly why a floor alone can't guarantee what actually gets installed. The gotcha I'd flag loudest: that urllib3 constraint is a floor, not a pin. 1.26.19 satisfies >=1.26.19 just as well as 2.7.0 does, so installing opensearch-py 2.8 doesn't guarantee your resolver picks a patched urllib3, it just stops forbidding it. Skip a separate urllib3>=2.5.0 pin in your own dependency file, and you can finish this entire migration still shipping the vulnerable version, because the old version still satisfies every constraint in the graph. We pinned it explicitly, with a comment citing the CVE, so the reason survives whoever edits that line next. # pyproject.toml [tool.poetry.dependencies] python = "^3.12" opensearch-py = "^2.8" # Explicitly floor urllib3 >= 2.5.0 for CVE-2025-50181 remediation. # opensearch-py only floors at >=1.26.19, so explicit flooring forces the resolver to upgrade. urllib3 = ">=2.5.0, 0 and bottom >= self.count: raise NotFound("Invalid page.") self.request = request self.page_number = page_number return list(self.response) def get_paginated_response(self, data): return Response({ 'count': self.count, 'facets': {}, # Maintained for strict frontend schema contract 'next': self.get_next_link(), 'previous': self.get_previous_link(), 'results': data }) By coupling this with an OrderingFilterBackend that transforms user query parameters (?ordering=-timestamp) into OpenSearch .sort({'timestamp': {'order': 'desc'}}) execution calls, we dropped django-elasticsearch-dsl-drf completely and uninstalled four transitive packages. Two Gotchas Worth Knowing Before You Try This An app-name collision broke an import at boot. opensearch-py's optional metrics module does from events import Events, expecting a small PyPI package called Events. Our own top-level Django app was also named events, and it shadowed the real package. The fix was a two-line shim satisfying the import, not a working metrics backend, just enough to stop the crash, since we don't use that feature anyway. "No product check" is the point, not a risk to route around. It's the entire reason one client works against both a legacy Elasticsearch-OSS container in CI and a real AWS OpenSearch domain in production. If you ever see UnsupportedProductError after this migration, it means a stray import elasticsearch survived somewhere. A full-tree grep should come back completely clean. Verification & Observability Strategy Validating search client migrations requires separating metadata aggregations (size=0) from real document retrieval hits. In Datadog, we isolated event retrieval traffic using query exclusions: service:backend-api @logger.name:opensearch -"size=0" A successful status 200 response on a non-size=0 query confirmed that both ordering backends, pagination slicing, and document mapping round-tripped cleanly through opensearch-py to AWS OpenSearch. Finally, running pip-audit verified the security milestone: $ pip-audit No known vulnerabilities found

Original Source

Read the full article at Hackernoon →

KhanList aggregates and links to publicly available news content. We do not host full articles from third-party sources. Always verify important information with original sources.