For a top-level executive summary of the Open Library project, please see the Main Open Library Index. This document contains a year-by-year breakdown of board reports, roadmaps, community call documents, a project index, and top-level team documents spanning engineering, design, communications, and more.
You can read more about the vision and mission of the Open Library here. You can also see a year-by-year breakdown of Open Library's roadmaps, meeting minutes, and other main documents here.
You can read our full year-by-year history here
As of 2024, the Open Library program is directed by Mek Karpeles and staffed by Drini Cami, Jim Champ, and Scott Barnes, with significant support from Lisa Seaberg from Patron Services. The project enjoys contributions from volunteers spanning more than 20 nations.
We drive most of our work on the Open Library project off GitHub Issues in the InternetArchive/OpenLibrary GitHub repo.
Much of our work is done by volunteers. So we plan very lightly, by year and by quarter, given the resources we have available. As new volunteers join, plans may change to accommodate their skills and interests.
Things we'd like to get done during a given year get the milestone for that year, e.g. "2019". For things we expect to be done in by the end of a given quarter, we apply the quarterly milestone, e.g. "2019 Q1" for things to get done on or before March 31, 2019.
Milestones are closed when their deadline arrives. Issues associated with that milestone that are not done get rescheduled (the milestone is changed) or backlogged (labeled State: Backlogged).
The assignee of an issue is the person responsible for its completion.
An example of a common search might be assignee:cdrini label:"state: work in progress" to see what Drini is working on.
Singular ownership is important to make sure things don't fall on the floor. We therefore avoid multiple assignees. Most issues have multiple individuals involved in various aspects of assessing and resolving it - those people are "mentioned" (e.g. "@hornc") in the issue comments.
If multiple folks are working together to solve a problem, use @mentions in the issue comments, or if it is really complicated, create a subissue to be owned by another person. Don't forget to mention the parent issue in the first comment.
Any open bug that is unowned is in need of triage.
We reserve a set of labels for use with Github issues, to assist with issue handling and project management for OpenLibrary.
Most generally, an issue evolves through a series of states, for example:
submitted -> assessed -> scheduled -> fixed -> closed.
On the way down that path, lots of things can happen. The labels below can be applied and removed during initial triage (either by the submitter, or by an initial triage), and thereafter by the owner of the bug.
The sections below contain:
Note the following:
The label's label (!) matters. Each label starts with a prefix that groups the labels into sorted list. Labels prefixed with "~" are deprecated. Labels prefixed with a developer's initials are managed by that developer (for their own purposes).
Label color matters. Each prefix has a common hue (developers, pick your hue!). Generally, the prefixes are orthogonal (an issue can only have one Priority, only one Type, from the managed set). So if you see an issue with more than label of a given hue, something might be awry. Labels get reduced color saturation (greyed out) labels to indicate they are deprecated. Lower color values (whiter) typically suggest reduced urgency. The label text also has these indicators, for accessibility (since not everyone sees all the colors).
Be careful touching labels on issues belonging to others! People are using the labels to actively manage their activity. You can add and remove labels from an issue:
a. At the time you submit the issue, as the submitter. b. The assignee/owner of an issue can label it as they see fit, using their own labels, and using managed labels according to the guidelines. c. The Bug Triage Owner will adjust issues during triage. d. Deputies will review and adjust labels as needed for issues in their area. e. If you're not sure, ask the issue owner, the Bug Triage Owner, @brad2014, or @mek. (The comment stream of the issue, or the slack channel are great places to raise these questions).
Every issue on Open Library's tracker should be assigned a Lead: @person and a Priority: # label by a member of staff.
Issues which have Needs: Lead and or Needs: Triage labels do not yet have a lead/mentor assigned and have not yet been vetted or evaluated for the community. Once an issue has been triaged, members of the community may request to be assigned to an issue and enjoy the mentorship of the designated Lead.
Priority describes how urgent the bug is. Very urgent bugs generally have an active conversation on the slack channel, so that they can be fixed right away.
When a priority label is applied to an issue by the submitter, or on any issue without an owner, it represents a suggestion, not a decision. Priorities are not immutable - even while an issue is being worked on, the owner may decide to move the priority up or down.
Although priorities indicate urgency rather than timing, a helpful frame for assigning priority is to ask questions such as these - think of them as "rules of thumb":
Is the issue preventing numerous users from successfully using the website? Is the issue related to leaking sensitive information? Is the issue related to usage or running processes actively corrupting our data? Does it really need to be fixed in the next 48 hours? Mark it Priority 0: Urgent. If it requires conversation or multiple developers to fix, they should all be talking in a slack channel right now.
Should a developer interrupt and set aside their current activity to get this issue resolved? Does it need to be fixed in the next 14 days? Assign an owner who will label it Priority 1: High. If you can't find a volunteer to own it, it can't be Priority 1.
The other priority labels are available to memorialize your assessment. They mostly indicate, "I've looked at this, and it is not urgent."
At most, one Priority label can be assigned to an issue. If there is no Priority label, consider the priority unassessed. It is good form to mark all your issues with some priority level, because it gives us a historical record of the distribution of issues by priority.
Every issue on Open Library must be assigned a lead label. A Lead is member of the community with domain expertise who has been appointed by staff to help manage a specific aspect of Open Library (such as search, design, javascript, i18n, etc). Anyone may apply to be considered for a specific lead position.
The Lead is responsible for:
See the Team Leads Labels to get an idea of who to tag.
Labels are grouped by prefix and color. If you create a label outside the managed set, prefix it with your initials and give your personal labels a common color. We are continually evolving the managed set to meet our needs. If you think a label deserves to be in the managed set, just mention it.
The labels are grouped into different axes for slicing and dicing issues:
What kind of issue this is. Is it something that is broken that should (perhaps) be fixed, or is it a request for a new feature or enhancement, or is it a reminder to reorganize or clean up some aspect of the code base?
Epics and subtasks are used when we want to separate out the ownership, comment stream, and timing of different parts of a large project. The Epic is closed when all its subtasks have been closed. For most issues, putting a checklist in the comment stream suffices (when everything is checked off, the issue can be closed). The "Needs: Breakdown" label can be used for any issue (epic or not) that needs a decision identifying the list of steps that will be taken in order to close the issue.
Note that Bug, Feature, Question, Refactor, Subtask are mutually exclusive. Every issue (post-triage) should have one of these. If an issue is labeled Epic, it probably also should have a Feature or Refactor label.
Use these labels to distinguish between issues that we're actively working on, those that we plan to work on, and those that seem to be good ideas that we'll consider when we have the additional time and resources required.
If no state label is present, the issue needs assessment.
If someone was working on an issue but had to set it aside, the state label might be changed to "Backlogged," or the current owner might find someone to hand it off to, or it might even be closed (if we decide it didn't need to be addressed after all).
If an issue is "State: Scheduled", it must have a milestone that indicates by when it is scheduled to be completed. We plan by quarters, so "2019 Q1" means it is an issue we expect to resolve on or before March 31, 2019.
If an issue is Priority 0 or Priority 1, and the state is not Work In Progress, something is wrong, and alarms should sound.
These labels indicate that an issue or pull request is stuck because the owner needs someone to respond - they'll add comments to the issue saying what exactly they need.
If you see one or more of these labels on an issue, assume we are not making progress on it.
If you are the owner of an issue and add this label, always add a comment that indicates, as best you can, what you need to get unstuck. If you think you need the help of another team member, make sure to mention them by their handle in the comment.
Remember to remove this label once the need is met and the issue is unstuck.
Issues typically lead to pull requests to modify the repo in order to resolve the bug.
It is considered good form, immediately prior to closing a bug, to add a label indicating if it was closed for any of the following reasons.
Some observations:
You'll almost always want to add additional detail in the comments as to how the decision to close was arrived at. Liberally mention other people you consulted with (for example, "I spoke to @jeff and we agreed this affects very few users.") and other issues (super-common: "this is a duplicate of issue #XXXX").
One of the best ways to attract attention to an issue that you feel is unfairly ignored is to close it prematurely.
If there is (or should be) another State: Scheduled issue to fix a big problem, and fixing the big problem would also resolve this issue, we have a choice: we could address this issue (e.g. with a stop-gap solution, right now, while we wait for the big fix), or close this issue (i.e. decide to leave it broken). In the comments, reference the big picture issue (e.g. "This issue will be properly resolved when we address issue #XXXX."), and explain your intentions. For example, you might comment "We'll leave this broken until then," and label Close: Will Not Fix, or comment "In the meantime, we'll deploy the following stop-gap" and label this issue, say, Priority 1: High.
There are some issues that affect multiple modules, or are related to a user story or workflow that touches multiple systems, and we use "theme" labels to identify them. This list is expected to grow.
The unifying characteristic of Themes is that they involve issues that touch many parts of the repo (UI, Server, Configuration, Documentation, Data).
It is expected that an issue will have at most one Theme: label.
The broad area this issue is related to, often suggesting who first should consider it.
It is preferred that an issue only have one Affects: label, but we're not religious about it. If you notice that an issue affects multiple areas in the above list, you may want to split it into multiple issues, one per area. If it makes sense to resolve them independently, that's enough. If they all need to be resolved in a coordinated fashion, create an Type: Epic issue which can remain open until all the subissues are closed.
These labels identify the specific module or service that the issue relates to. Often this corresponds to a particular directory or file or interface or class present in the repo hierarchy. This list is expected to grow.
A common search might be something like label:"Affects: Server" label:"Module: Solr" label:"State: Work In Progress" to see who is actively working on calls to solr in the server. If you wanted to pick up issues in that area, you could see who else is doing so.
A few remaining labels that are not in any group, because of github conventions, or for other reasons.
Color Label Description Good First Issue Easy issue. Good for newcomers. [managed]A good first issue should be clear, should not require a lot of context, should be low risk and easy to review. Issues that involve high priority or global changes to the production system code are not good candidates.
If you have a large number of issues assigned to you, there may be times when you want to divide them into groups by your own criteria. You can do this by creating your own labels. They all go into the repository label set, so we ask that you:
Pick your own unique (!) color that you apply to all your personal labels, and
Prefix the label name with your initials.
For example, Charles Horn creates his own labels with prefix CH: and color #1d76db.
Labels that are grey and/or start with a tilde ~ are deprecated. They typically are not used much, and shouldn't be added to issues going forward.
The default issues page presents several search fields. There is one at the very top, for searching the repository or all of github. There is another one under the issues tab that searches through the issues. To the right of the issues filter is the labels button, which gives you a "labels list" to browse and drill down to issues.
You can type a label search term directly into the issues filter search field, you type label:<labelname> to include issues containing the label, and -label:<labelname> to exclude issues containing that label. Because all the managed label names contain spaces and colons, you need to quote them if you are typing them directly into the issue search bar. For example, label:"affects: documentation" (case doesn't matter).
If the issues filter field contains multiple labels terms, you will see the issues that match all those label terms. There is no way to see a list issues that contain label A or label B.
If you're not sure exactly what the label is, just use the labels pull down, and in its search bar type any word that is in the label name or description, and the matches will be displayed. Click on the labels you want to add to your issues filter.
All the managed labels have [managed] in their description, so an easy way to browse just the managed labels is to search labels on that string.
It is easy (and sometimes dangerous) to do batch updates of labels (or milestone, or assign, or close). Just filter the issues to the ones you want to change, select the issues you want to label. When you select one or more issues, the pulldown menu changes from filter mode to update mode. Pull down the label menu, and you can select which labels you want to add or remove (it's a toggle).
Leads have the challenging job of monitoring and keeping up with progress on their issues and pull requests. Staff has several bots that we hope can make the process easier.
For each of the following, use the labels facet to add your label to see issues and PRs under your leadership:
labels facet to add your label to see issues you need to triage.labels facet to add your label to see issues you need to triage.If you don't have merge permissions and a PR looks ready to go, please mark it is "Needs: Staff / Internal".
We use assignee to denote PR ownership. If you are the assignee, then you should have the PR on your todo list until you merge or close it.
The assignee of a PR is responsible for:
Needs: Author Input or Needs: Review as necessary.Needs: Deploy Approval and get that approval before merging.Each Monday (as of 2022) we triage PRs (excluding drafts) and make sure they have leads assigneed so that nothing gets stuck.
Frequently Asked QuestionsDon't see what you're looking for? Check questions asked by contributors on Github or submit your own question
Topics which could still use recipes
See: https://openlibrary.org/dev/docs/api/search
Currently Logged-in Patron Adding new db table schema to Open LibrarySee: https://github.com/internetarchive/openlibrary/pull/7928
Setting / unsetting a Cookie on login or registrationYou might want to set a cookie when a patron logs in or registers to persist information, such as a setting, across page views. For instance, if a patron logs in and they have print disabled access (something we'd need to check on every page view) we can save this using a session cookie:
web.setcookie('sfw', 'yes', expires="")
Setting cookies example: https://github.com/internetarchive/openlibrary/pull/7935/files#diff-ef23faa0d3112e7eef1f574a58b0710f9604f911f5a57e795e45c13c4b91fa19R342
Clearing cookies example: https://github.com/internetarchive/openlibrary/pull/8490/files#diff-884f3a712a7e51cce625ca19060335684272f33a967702b19e66e48dbdd1be69R131-R133
Magic incantation: Accessingweb.ctx
Need to test Open Library functionality from the command line in python? You can use this magic incantation to load the minimal Open Library app to test models, use web.ctx.site to fetch data, and more.
First, you need to exec into the docker container and launch python: docker compose exec -it web python
Next, use the following incantation to load Open Library and launch a minimal headless app:
import web
import infogami
from openlibrary.config import load_config
# load_config('/olsystem/etc/openlibrary.yml') # if production
load_config('config/openlibrary.yml') # if local
infogami._setup()
from infogami import config
from openlibrary import accounts logged_in_user = accounts.get_current_user()
from openlibrary import accounts
from openlibrary.accounts.model import OpenLibraryAccount
logged_in_user = accounts.get_current_user()
username = logged_in_user and logged_in_user.key.split('/')[-1]
account = username and OpenLibraryAccount.get(username=username)
s3_keys = web.ctx.site.store.get(account._key).get('s3_keys') Testing the affiliate API
import web
import infogami
from openlibrary.config import load_config
load_config('/olsystem/etc/openlibrary.yml')
infogami._setup()
from infogami import config
from openlibrary.core.vendors import AmazonAPI
web.amazon_api = AmazonAPI(*[config.amazon_api.get('key'), config.amazon_api.get('secret'),config.amazon_api.get('id')], throttling=0.9)
book = web.amazon_api.get_products(["1666568287"], serialize=True) Examine the affiliate queue
ol-home0docker exec -it openlibrary-affiliate-server-1 bashcurl localhost:31337/statusOr to monitor repeatedly during debugging:
% cat > ~/affiliate_status.sh
#!/bin/bash while true do curl --silent localhost:31337/status | cut -c 1-90 sleep 1 done
The Internet Archive’s Open Library Fellowship is a flexible, self-designed independent study which pairs volunteers with mentors to lead development of a high impact feature for OpenLibrary.org.
Most fellowship programs last one or more months and are flexible, according to the preferences of contributors and availability of mentors. We typically choose fellows based on their exemplary and active participation, conduct, and performance within the Open Library community. The Open Library staff typically only accepts 1 or 2 fellows at a time to ensure participants receive plenty of support and mentor time.
List of Fellowship Opportunities
Occasionally, funding for fellowships is made possible through Google Summer of Code or Internet Archive Summer of Code & Design.
If you’re interested in contributing as an Open Library Fellow and receiving mentorship, you can apply using this form or email openlibrary@archive.org for more information.
RetroSearch is an open source project built by @garambo | Open a GitHub Issue
Search and Browse the WWW like it's 1997 | Search results from DuckDuckGo
HTML:
3.2
| Encoding:
UTF-8
| Version:
0.7.4