YipYip
2020
Landing page • Chrome Web Store • Github Repo
YipYip started as a shower thought about wanting more from find-in-page/CMD + F. Sure, word search is useful, but I wanted something more powerful. I wanted help "finding" things on a page other than just text. I wanted to jump around webpages, especially ones that I visited often and frequently performed the same actions on (eg. deleting emails on Gmail or hopping between the tabs of a Github repo).
Why couldn't the find-in-page menu be my command center for the internet?
Keyboard Navigation
I decided to focus first on using a find-in-page like experience for keyboard navigation. Many websites support the use of hot keys as shortcuts to navigate and perform actions. However, these shortcuts differ greatly between sites, are a significant undertaking to learn, and of course, not every website has them.
Meanwhile, though invisible to most non-developers, webpages contain tons of contextual information about interactive elements on the page like buttons, links, and input fields. In most cases, these elements and their contextual information roughly correlate to the actions that are available to the user to perform. Using this invisible information, I could allow a web user to perform actions, click buttons, and highlight inputs, by simply typing the name of the action they wanted to perform.
For example, if a user wants to delete an email in Gmail, instead of taking the time to move their mouse to the trashcan icon, they can simply start typing "d", "e", "l" (the start of "delete") and YipYip will automatically highlight the trash icon, allowing them to click "Enter" to delete the email.
To the average web user, Gmail's toolbar just consists of icons. However, the webpage actually contains contextual information labelling the DOM elements. For example, the button with a trash icon shown here has data-tooltip and aria-label describing the button's use.
Getting semantic
In addition to matching DOM elements against the user's exact query, YipYip also matches against synonyms of the user's query. This helps in the case that a user describes their intention in a slightly different way than the webpage does (trash vs. delete vs. discard), especially for buttons which only display an icon and no text.
We explored the idea of using a precompiled synonym library or a word similarity algorithm, like word2vec but decided against such a "generalized" solution because the meaning of a link or button on a webpage is extremely context dependent. Instead, YipYip has a configuration file per URL host with synonyms specific to that host. Each configuration file can be thought of as mapping to an "App" used in the browser (eg. mail.google.com for Gmail, news.ycombinator.com for Hacker News).
Introduction
Last Mile Search
I like to think of YipYip as an experiment in what I call "last mile search".
Last Mile Search is when a search engine (Google) is good enough to get you to a web page (or file on desktop, collection of documents, etc.) that’s highly likely to contain the information you need but wasn’t able to infobox an answer or artificially generate an answer on the results page. The primary solutions to this problem right now are to either scan the entire page manually or do find-in-page for an exact word match.
Some alternate solutions include:
- fuzzy search (matching misspelled terms)
- semantic “similar” word search (synonyms)
- semantic Q&A search
- automatically generated answer based on previous google/search engine search applied to chosen page/document
Microsoft Edge and Arc have both recently launched a version of AI search to augement find-in-page. Egde does a combination of options 1 and 2 above by suggesting related matches and synonyms to a user's query even if they misspelled a word. Meanwhile, Arc opted for 3, a more full fledged AI search experience in which you can type a question and an LLM will generate a response using the webpage as context. 4 feels like something Google Chrome should already do...
The Future
I think it would be great to add multiple modes to YipYip so that users can use a keyboard shortcut to hop between three options based on context, potentially with a default mode per site:
- keyboard navigation (existing functionality)
- fuzzy & synonym based find-in-page
- AI Q&A (as a paid feature or paste-your-own OpenAI token)
Find-in-page is a prime example of how browsers have stagnated for 20+ years and are ripe for innovation. Please reach out if you'd like to contribute to adding these web browsing super powers to all browsers!
P.S.
Yes, for you Avatar fans out there, YipYip is named after what Aang yells to his huge flying bison/manatee, Appa, to get it to take off into the air. As an equivalent to "Giddy up!", we felt this phrase signifies YipYip's ability to help you quickly jump into action on the web.
