Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 9 additions & 5 deletions design/components/json-ast.md
Original file line number Diff line number Diff line change
Expand Up @@ -92,11 +92,15 @@ status with it:

## Locations

- A node's location is a URI whose fragment is a JSON Pointer. Pointer
segments are percent-encoded where a fragment requires it
(`appendToLocation`, `@hyperjump/uri`'s `encodeIriFragment`), so
locations are valid, normalized IRIs. `get` decodes a fragment before
following it as a JSON Pointer.
- A node's location is a URL whose fragment is a JSON Pointer. Pointer
segments are percent-encoded with the URL fragment percent-encode set
(`appendToLocation`), so a location is what a URL parser makes of it and
can be compared to a resolved reference. `%` is encoded too, because
`get` decodes a fragment before following it as a JSON Pointer.
- The set is encoded directly instead of by a URL parser, because a parser
removes tabs and newlines.
- A lone surrogate in a property name is encoded as U+FFFD, as a URL
encodes it, so two property names can have the same location.
- `appendToLocation` caches the encoded form of each segment, because
encoding was most of the time `fromJs` took (see
[Node locations](../benchmarks/node-locations.md)).
2 changes: 1 addition & 1 deletion design/principles/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,6 @@ The rules decisions are made by.

- [A generic client](generic-client.md)
- [`retrieve` is only for retrieving](retrieve-is-for-retrieving.md)
- [Retrieval uses URLs](retrieval-uses-urls.md)
- [URLs, not URIs](urls-not-uris.md)
- [Documents aren't trusted](documents-arent-trusted.md)
- [Parsing never fails](parsing-never-fails.md)
6 changes: 0 additions & 6 deletions design/principles/retrieval-uses-urls.md

This file was deleted.

30 changes: 30 additions & 0 deletions design/principles/urls-not-uris.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,30 @@
# URLs, not URIs

Browser is a WHATWG client, so what it identifies is a WHATWG URL. A
reference's `$ref` is resolved with `new URL`, what's retrieved is a URL
without its fragment, and a node's location is a URL that a URL parser
leaves as it is. `fetch`, browsers, and Node.js already agree on what a URL
is, so Browser doesn't need a URI library or rules of its own.

- A reference is resolved the way a browser resolves a link. It's
percent-encoded where a URL requires it, so a `$ref` with non-ASCII
characters resolves to its encoded form. A host is lowercased, and a
special scheme's host is converted to ASCII.
- Applications that identify documents by URIs that aren't URLs handle
that themselves.

Rejected:

- Resolving references by RFC 3986 with `@hyperjump/uri`. It resolves a
relative reference against any scheme and keeps IRIs as they are. But
retrieval uses URLs, so a session would have two ideas of what an
identifier is. A reference resolved by one and retrieved by the other
can be a different resource, as with a `%2e%2e` segment, which a URL
treats as `..` and RFC 3986 doesn't. It's also a dependency for what the
platform already does.

Known limitations:

- A relative reference can't be resolved against a URL with an opaque path,
like `urn:example:a`. Only a fragment can. Parsing a document with one
throws.
17 changes: 3 additions & 14 deletions package-lock.json

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

1 change: 0 additions & 1 deletion package.json
Original file line number Diff line number Diff line change
Expand Up @@ -81,7 +81,6 @@
},
"dependencies": {
"@hyperjump/json-pointer": "^2.0.0",
"@hyperjump/uri": "^1.4.0",
"@types/unist": "^3.0.3",
"content-type": "^3.1.1",
"http-cache-semantics": "^4.3.0",
Expand Down
4 changes: 2 additions & 2 deletions src/hyperjump/browsing.js
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
import { toAbsoluteIri } from "@hyperjump/uri";
import { jsonArrayIter, jsonObjectEntries, jsonObjectHas, jsonObjectKeys, jsonObjectValues, jsonStep, jsonValue } from "../json/jsonast-util.js";
import { withoutFragment } from "./urls.js";

/**
* @import { JrefNode } from "../jref/jref-ast.d.ts"
Expand Down Expand Up @@ -57,7 +57,7 @@ export const jrefValue = /** @type API.jrefValue */ ((/** @type JrefNode */ node
*/
const follow = async (session, node) => {
if (node.type === "reference") {
return session.get(node.href, { referencedFrom: toAbsoluteIri(node.location) });
return session.get(node.href, { referencedFrom: withoutFragment(node.location) });
} else {
return node;
}
Expand Down
20 changes: 14 additions & 6 deletions src/hyperjump/file-uris.js
Original file line number Diff line number Diff line change
@@ -1,4 +1,3 @@
import { parseIri, resolveIri, toRelativeIri } from "@hyperjump/uri";
import { contextUri } from "#context-uri";

/**
Expand All @@ -12,15 +11,24 @@ export const relativeFileUri = (uri) => {
return uri;
}

// Parsing it normalizes it, so a scheme written as 'FILE:' is relative too
const url = new URL(uri);
const suffix = url.search + url.hash;

// There are no files to be relative to where the working directory isn't
// a folder, like in a browser
const base = contextUri();
if (!/^file:/i.test(base)) {
const { path, fragment } = parseIri(uri);
const name = path.slice(path.lastIndexOf("/") + 1);
return fragment === undefined ? name : `${name}#${fragment}`;
return url.pathname.slice(url.pathname.lastIndexOf("/") + 1) + suffix;
}

// The working directory's URL ends with a `/`, so its last segment is empty
const directories = new URL(base).pathname.split("/").slice(0, -1);
const segments = url.pathname.split("/");
let common = 0;
while (common < directories.length && common < segments.length - 1 && directories[common] === segments[common]) {
common++;
}

// Resolving it normalizes it, so a scheme written as 'FILE:' is relative too
return toRelativeIri(base, resolveIri(uri, base));
return "../".repeat(directories.length - common) + segments.slice(common).join("/") + suffix;
};
2 changes: 1 addition & 1 deletion src/hyperjump/get.test.js
Original file line number Diff line number Diff line change
Expand Up @@ -195,7 +195,7 @@ describe("JSON Browser", () => {
const subject = await session.get(`${testDomain}/foo#/a%5Eb/c%20d`);

expect(toJson(subject)).to.equal(`42`);
expect(subject.location).to.equal(`${testDomain}/foo#/a%5Eb/c%20d`);
expect(subject.location).to.equal(`${testDomain}/foo#/a^b/c%20d`);
});

test("an unnecessarily encoded fragment finds the same node", async () => {
Expand Down
28 changes: 14 additions & 14 deletions src/hyperjump/hyperjump.js
Original file line number Diff line number Diff line change
@@ -1,4 +1,3 @@
import { parseIri, resolveIri, toAbsoluteIri } from "@hyperjump/uri";
import { contextUri } from "#context-uri";
import { defaultUriSchemePlugins } from "#default-uri-scheme-plugins";
import { JsonMediaTypePlugin } from "./media-types/json-media-type-plugin.js";
Expand All @@ -7,6 +6,7 @@ import { jsonPointerGet } from "../json/jsonast-util.js";
import { acceptHeader, fragmentPointer, mediaTypeByExtension, mediaTypePluginFor } from "./media-type-plugins.js";
import { AllowedUrls, configSource, resolveAllowRetrieval, sessionSource } from "./allowed-urls.js";
import { relativeFileUri } from "./file-uris.js";
import { resolveUrl, withoutFragment } from "./urls.js";

/**
* @import { JrefNode } from "../jref/jref-ast.d.ts"
Expand Down Expand Up @@ -189,29 +189,29 @@ export class Session {
* @returns {Promise<FollowedNode>}
*/
async #get(uri, options, referenceChain) {
uri = resolveIri(uri, this.#contextUri);
uri = resolveUrl(uri, this.#contextUri);
if (referenceChain.includes(uri)) {
throw new ReferenceCycleError([...referenceChain, uri]);
}
referenceChain = [...referenceChain, uri];

const { fragment } = parseIri(uri);
const { hash } = new URL(uri);
// Subclasses can override getDocument, so every lookup goes through it
const document = await this.getDocument(uri, options);

const cursor = fragment === undefined ? "" : this.#hyperjump.fragmentPointer(document, fragment);
const cursor = hash === "" ? "" : this.#hyperjump.fragmentPointer(document, hash.slice(1));
const node = jsonPointerGet(cursor, document.children[0]);
if (node.type === "reference") {
return this.#get(node.href, { referencedFrom: toAbsoluteIri(node.location) }, referenceChain);
return this.#get(node.href, { referencedFrom: withoutFragment(node.location) }, referenceChain);
} else {
return node;
}
}

/** @type API.Session<JrefNode>["getDocument"] */
async getDocument(uri, options = {}) {
uri = resolveIri(uri, this.#contextUri);
const id = toAbsoluteIri(uri);
uri = resolveUrl(uri, this.#contextUri);
const id = withoutFragment(uri);

this.#cache[id] ??= this.#retrieveDocument(uri, options).catch((/** @type unknown */ error) => {
// A failure isn't cached, so it can be tried again
Expand All @@ -234,12 +234,12 @@ export class Session {
* @type (uri: string, options: API.GetOptions) => Promise<DocumentNode>
*/
async #retrieveDocument(uri, options) {
const id = toAbsoluteIri(uri);
const id = withoutFragment(uri);

let response;
let url = uri;
for (let redirects = 0; ; redirects++) {
const hop = toAbsoluteIri(url);
const hop = withoutFragment(url);
if (!await this.#allowedUrls.allows(hop)) {
throw this.#notAllowedError(id, hop);
}
Expand All @@ -263,13 +263,13 @@ export class Session {
// `retrieve` gives every response a URL. The fragment isn't part of
// what's retrieved.
try {
url = toAbsoluteIri(resolveIri(location, response.url));
url = withoutFragment(new URL(location, response.url).href);
} catch (error) {
throw new RetrievalError(uri, { cause: error });
}
}

if (response.url !== toAbsoluteIri(url) && !await this.#allowedUrls.allows(response.url)) {
if (response.url !== withoutFragment(url) && !await this.#allowedUrls.allows(response.url)) {
await response.body?.cancel();
throw this.#notAllowedError(id, response.url);
}
Expand All @@ -280,7 +280,7 @@ export class Session {
// References within the document are relative to the URI it was
// retrieved from, which can differ from the requested URI (redirects,
// symlinks), so it needs to be cached under both.
this.#cache[toAbsoluteIri(response.url)] ??= Promise.resolve(document);
this.#cache[withoutFragment(response.url)] ??= Promise.resolve(document);

return document;
} catch (error) {
Expand All @@ -298,7 +298,7 @@ export class Session {
* @type (uri: string, options: API.GetOptions) => Promise<Response>
*/
async #retrieve(uri, options) {
const { scheme } = parseIri(uri);
const scheme = new URL(uri).protocol.slice(0, -1);

const plugin = this.#hyperjump.uriSchemePlugins.get(scheme);
if (!plugin) {
Expand All @@ -312,7 +312,7 @@ export class Session {
mediaTypeByExtension: (uri) => mediaTypeByExtension(mediaTypePlugins, uri)
});
if (!response.url) {
Object.defineProperty(response, "url", { value: toAbsoluteIri(uri) });
Object.defineProperty(response, "url", { value: withoutFragment(uri) });
}

return response;
Expand Down
8 changes: 3 additions & 5 deletions src/hyperjump/uri-schemes/file-scheme-plugin.js
Original file line number Diff line number Diff line change
@@ -1,10 +1,10 @@
import { createReadStream } from "node:fs";
import { realpath } from "node:fs/promises";
import { fileURLToPath, pathToFileURL } from "node:url";
import { parseIri, toAbsoluteIri } from "@hyperjump/uri";
import { Readable } from "node:stream";
import { FileNotFoundError } from "../hyperjump.js";
import { relativeFileUri } from "../file-uris.js";
import { withoutFragment } from "../urls.js";

/**
* @import * as API from "./file-scheme-plugin.d.ts"
Expand All @@ -19,9 +19,7 @@ export class FileUriSchemePlugin {
/** @type API.FileUriSchemePlugin["retrieve"] */
async retrieve(uri, options) {
if (options.referencedFrom) {
const { scheme } = parseIri(options.referencedFrom);

if (scheme !== "file") {
if (new URL(options.referencedFrom).protocol !== "file:") {
throw Error(`Accessing a file (${relativeFileUri(uri)}) from a non-filesystem context (${options.referencedFrom}) is not allowed`);
}
}
Expand All @@ -43,7 +41,7 @@ export class FileUriSchemePlugin {
}
throw error;
}
const responseUri = realFilePath === filePath ? toAbsoluteIri(uri) : pathToFileURL(realFilePath).toString();
const responseUri = realFilePath === filePath ? withoutFragment(uri) : pathToFileURL(realFilePath).toString();

// A file doesn't have a media type, so it's the one its extension has
const headers = new Headers();
Expand Down
6 changes: 3 additions & 3 deletions src/hyperjump/uri-schemes/http-scheme-plugin.js
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
import CachePolicy from "http-cache-semantics";
import { toAbsoluteIri } from "@hyperjump/uri";
import { defaultHttpCache } from "#default-http-cache";
import { redirectMode } from "#redirect-mode";
import { withoutFragment } from "../urls.js";

/**
* @import { HttpCacheStore } from "./http-cache-store.d.ts"
Expand Down Expand Up @@ -86,12 +86,12 @@ export class HttpUriSchemePlugin {
const headers = new Headers(response.headers);
headers.set("content-type", mediaType);
const typedResponse = new Response(response.body, { status: response.status, statusText: response.statusText, headers });
Object.defineProperty(typedResponse, "url", { value: response.url || toAbsoluteIri(uri) });
Object.defineProperty(typedResponse, "url", { value: response.url || withoutFragment(uri) });

return typedResponse;
}

const url = toAbsoluteIri(uri);
const url = withoutFragment(uri);
const key = JSON.stringify([url, accept]);
let pending = this.#pending.get(key);
if (!pending) {
Expand Down
18 changes: 18 additions & 0 deletions src/hyperjump/urls.js
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
/**
* Resolve a URL against the context URI. Where there's no context URI, as
* in a browser without a document, only an absolute URL can be resolved.
*
* @type (url: string, base: string) => string
*/
export const resolveUrl = (url, base) => new URL(url, base === "" ? undefined : base).href;

/**
* The URL of what's retrieved, which is the URL without its fragment
*
* @type (url: string) => string
*/
export const withoutFragment = (url) => {
const parsed = new URL(url);
parsed.hash = "";
return parsed.href;
};
3 changes: 1 addition & 2 deletions src/jref/jref-util.js
Original file line number Diff line number Diff line change
@@ -1,4 +1,3 @@
import { resolveIri } from "@hyperjump/uri";
import { fromJson, hasStatus as jsonHasStatus } from "../json/jsonast-util.js";

/**
Expand Down Expand Up @@ -40,7 +39,7 @@ const annotateReferences = (node, uri) => {
Object.assign(node, {
type: "reference",
// Without a base URI, relative references can't be resolved
href: uri === undefined ? valueNode.value : resolveIri(valueNode.value, uri)
href: uri === undefined ? valueNode.value : new URL(valueNode.value, uri).href
});
isReference = true;
}
Expand Down
Loading
Loading