How should Papers export/import BibTeX libraries?

jowens's Avatar

jowens

24 Mar, 2011 03:56 AM

As a frequent paper author and reviewer, I take my BibTeX quite seriously. Papers 2.0 is the first version that is a good enough candidate for exporting bibtex. I've been sending boatloads of suggestions to the Papers developers in an effort to improve it. They asked that we discuss bibtex export needs here to try to get a consensus. I haven't dealt with bibtex import, but if that's relevant, let's discuss it here too.

Here's some issues that I definitely urge the developers to fix:

  • Titles are wrapped in {{}}. They need to be wrapped in only one {}. It would be desirable if it was possible to mark characters as always capitalized, so they could get wrapped in their own {}. Example:

    Current: title = {{Using GPUs to Improve Multigrid Solver Performance on a Cluster}},

    Should be: title = {Using {GPU}s to Improve Multigrid Solver Performance on a Cluster},

    "publisher" is also wrapped in {{}}, which is totally unnecessary since it's ignored anywhere except "title".

  • Months should always be three-letter abbreviations not wrapped in anything, since those abbreviations are built into bibtex and then translate properly to other languages if necessary (and allow the bib style to select if it wants Sept., September, 9, etc.).

    Current: month = {sep},

    Should be: month = sep,

    Ideally, Papers should also support month/month (month = mar # "\slash " # apr) and day-month (month = "2~" # dec).

  • @incollection has multiple issues: no way to mark the book title (what exports as "publisher" is what I would expect to be "booktitle"); no way to mark the chapter number; no way to mark the book volume number.

  • Putting parentheses in an author name is broken. Poor "Deng, Yangdong (Steve)" gets corrected to "{Deng, Yangdong {Steve}" (note the braces, and the lack of a closing brace).

  • url is currently broken, since all fields have to be either numbers, defined abbreviations, wrapped in {}, or wrapped in "", and since it's wrapped in \url{} not {\url{}}, it breaks bibtex. See below for more on this though (what is the right thing to do).

  • Any time two initials are exported, they need to be separated. "Bethel, EW" means poor Wes's first name is "EW", which it's not. "Bethel, E.W." is also not right - separate the E. and W. with a space.

  • Support both @phdthesis and @mastersthesis.

Things I'd like:

  • In general, importing from ACM and IEEE into bibtex loses both first names (instead replacing them with initials) and periods after initials.

  • Customizable citation formats (my field uses Lastname:YYYY:ABC, where ABC are the first three letters in the first three words in the title)

  • Fixing ACM and IEEE exports (capitalization in venue titles, weird formatting of venue titles, making the exported title match the capitalization of the title on the paper, etc.)

  • If the URL is actually a DOI, put that in the "doi" field instead. Right now Papers is (correctly) coding these as just the right half of the DOI (no http://, no dx.doi.org).

Things I'm not sure about:

  • Formatting of urls. Right now the export uses url = \url{http://...}. I know that won't work because everything has to be wrapped in a {} or a "", so "\url{http://...}" would make more sense. My feeling is that we should NOT be asking for \url{} though, instead just citing the URL without wrapping it, since most bib styles that use the url field will wrap it themselves. But this is a good topic for discussion.

  • Formatting of local_url. Right now these have %s in the titles, which appears to work correctly on my latex installation, which is good. Nice to get feedback on others' setups though. My current thought is that url should be formatted exactly as local_url is now.

There are other things that will surely bug me once I actually get started using this export to write papers. Anyway, hoping that we can get some good discussion going on what the right thing to do is so we can give guidance to the Papers developers.

  1. 122 Posted by jowens on 17 Feb, 2012 08:26 PM

    jowens's Avatar

    @jody.foo: I would respectfully submit that putting braces around every capital letter in certain fields is not the right thing to do. If you want your bibliographies to capitalize everything in those fields, change the bib style, not the bibliography. Braces in the bibliography should be reserved for only letters in the title that must be capitalized no matter what capitalization is applied to the title as a whole (i.e. {E}instein).

    I am also aware of no bib styles where braces are required except for "title" (I've never seen a bib style change the bibtex's capitalization of publisher, booktitle, or journal).

  2. 123 Posted by klaxamazoo on 19 Feb, 2012 09:40 PM

    klaxamazoo's Avatar

    Has the problem with the extra set of {} in the title of the .bib export been fixed yet?

  3. 124 Posted by Lasse on 02 Mar, 2012 11:12 AM

    Lasse's Avatar

    this regards the @incollection type:
    I love the way one can drag a paper onto a book to make it a chapter.
    It would be great if Papers exported the editors of the book into the incollection entry.

  4. 125 Posted by Roz on 08 Mar, 2012 04:52 AM

    Roz's Avatar

    Will an automatic .bib be implemented with the next update (would be great if in preferences we could set the path and file name for our automatic .bib output) and if a button could be added to the menu at the bottom to request a sync (with all devices and libraries specified).
    Thanks for the great work!!!

  5. 126 Posted by mywoodstock on 20 Mar, 2012 05:58 AM

    mywoodstock's Avatar

    +1 for .bib generation.

    Also a way to give custom citekeys? E.g. if I do not want the colon in {Smith:2009ac}, or I only want the last two digits of the year?

  6. 127 Posted by nicofournier.volcano on 21 Mar, 2012 01:17 AM

    nicofournier.volcano's Avatar

    I found the BibTex export much, much better in Papers2 (vs. Papers1).

    About what could be improved: another pair of thumbs up for the automatic .bib generation. It would drastically streamline my workflow and avoid the back-and-forth between latex and .bib export from Papers when adding a missing ref.
    Some suggestions: when I write a paper, I usually set up a specific, dedicated collection where I dump all the relevant references (plus quite a few more) that I may use for that paper. The .bib automatic generation could be done everytime an item is added to a specific collection. Filename for the .bib could even be that of the collection. If this is too cumbersome, another option could be to have a centralized place (standard or customizable?) where .bib's could be generated for each collection (or even the whole library if the "collection route" is a no-go).
    Personally, I avoid using a single .bib for my whole library due to the shear number of references in it. But some people may have a different view.

  7. 128 Posted by Aldaron on 28 Mar, 2012 02:44 AM

    Aldaron's Avatar

    I'm confused about Papers's BibTeX support. In effect, it doesn't seem to support it at all, except in an extremely limited sense. As near as I can tell, for example, there's no support, for eprints, patents, or for many other standard types and fields required for any reasonable deployment of BibTeX. Papers seems to treat everything as either a "book", an "article" or as "misc". How do I ensure that patents are exported as patents, eprints (such as arXiv articles) as eprints (with eprinttype={arxiv} and, e.g., eprint={0906.1204})? What am I missing?

  8. 129 Posted by klaxamazoo on 28 Mar, 2012 05:28 PM

    klaxamazoo's Avatar

    Nope, it doesn't look like the problem with an extra set of {} being added in the Title has been fixed.

  9. 130 Posted by Jean-Luc on 29 Mar, 2012 02:45 PM

    Jean-Luc's Avatar

    How does the bibtex import command work? I created a simple .bib file (below) the tried several ways: from menu, by dragging, etc. Each time there's a brief "flash" and I'm left in the Recent Import section, with no import listed. I verified that the paper has not been added.

    Thanks,

    jL

  10. 131 Posted by Thomas Hodgson on 07 May, 2012 10:39 AM

    Thomas Hodgson's Avatar

    As with many people I would be very happy if Papers2 were able to keep a .bib file in sync with the citations I put in with Manuscripts. It would be almost as good if there were a way to select and then export all the papers that are listed as citations when I look at a file in My Research\Manuscripts. Being able to have a (smart) collection for each one would also be nice.

  11. 132 Posted by Marlina Hoffmann on 24 May, 2012 10:19 AM

    Marlina Hoffmann's Avatar

    Hi,

    I'm just starting to use Papers and LaTeX both and now discovered the package biblatex which also allows to give a shortjournal-title like "Hum. Mol. Genet.", short for "Human Molecular Genetics" (I'm not sure if this is also available in BibTeX). It would be really helpful if Papers would export that when saving the library as BibTeX.

    And in generell it would be nice too have automatical saving for .bib format, everytime you add a journal instead of having to save manually.

    Keep up the good work :)
    Marlina

  12. 133 Posted by Pif Edwards on 24 May, 2012 05:28 PM

    Pif Edwards's Avatar

    I strongly agree with this.

    The most pressing issue for me is conference proceedings are exporting very badly and references/notes exported (or dragged) are not usable. I am getting publisher where I should be getting proceedings title. I'd also like the date and city exported. Conferences are king in my field, this is kind of a big deal.

    This is a must have for me.
    Pif

  13. 134 Posted by andrew on 25 May, 2012 06:55 AM

    andrew's Avatar

    Not sure if it fits within the context of this discussion, but I'd like to see Manuscripts able to cope with TeX-style documents, which often consist of multiple files, and be able to detect (and thus export) which citations are used within the Manuscript.

    The Manuscripts feature can be used to insert references, even in an editor like MacVim, and this places the file in your Manuscripts within Papers2, but it doesn't cope with a multi-tex-file document, nor does it detect existing citations within the document (only those you insert using Papers2's insert citation feature).

    I'm not sure if its common for non-TeX systems to have multi-file documents, but having multiple files in a Manuscript (either as Chapters or Supplements) doesn't seem to be useful. BibTeX citations can only be inserted into the main file (in supplement files the insertion is just silently ignored), so you'd effectively need to have every single .tex file as a separate document. I'm not sure if extra files should be "supplements" or "chapters" or what — they're not really either! — so perhaps the right thing to do would be for Papers2 to scan for (recursive!) \include{} and \input{} commands (this could get complex with more than just a few different markup languages?), or the ability to manually specify a set of files (or directories) as the Manuscript source? I generally have a separate folder for everything pertaining to one manuscript, which may have many sub-folders for extra material.

    Papers2 also seems to struggle with different filetypes in the Manuscript. I added a PDF to my thesis Manuscript as the primary document, and it now attempts to view that in MacVIm instead of within Papers itself when I double click on the Manuscript. So that's obviously not the intention! The preview window does accurately show the PDF preview though...

    Scanning for existing citations in a document could improve the list of citations for the Manuscript, and thus allow it to automatically export just the required citations for a .bib file — right now the Citations list is limited to just those that were inserted with the Insert Citation feature, but that doesn't account for removals, nor existing citations, or e.g. those in copied text. The .bbl file (filename otherwise matches the master .tex file) could be leveraged to get a list of citations (from the last time it was formatted), or a recursive scan of the master .tex file + \include and \input commands would achieve the same with more up-to-date info (but would be harder to ensure you are definitely including the right files, as full/relative paths may mean different things in different configurations).

    Finally, even though Papers2 has a concept of a "formatted" document vs the source for Manuscripts (which would be PDF vs .tex in my case), the formatted doc doesn't seem to be treated as anything more than a supplement. It seems to me like Papers should show both source and formatted docs for systems that use BibTeX citations, since the formatting is obviously being performed by external software, so the workflow isn't controlled by Papers2.

  14. 135 Posted by papers.michael on 06 Jun, 2012 03:46 PM

    papers.michael's Avatar

    I had assumed that applying formatting like italics when editing a title in Papers would be exported as \emph{} by the bibtex exporter, but it doesn't seem to be.

    Is this right, and if so could this feature be added to a future release?

    Cheers

  15. 136 Posted by bruce.hunt on 10 Jun, 2012 06:25 AM

    bruce.hunt's Avatar

    In response to papers.michael....
    My view is that formatting of an text entered into Papers would be stripped off on export to BibTeX or any of its derivatives/variants.
    The whole concept of using TeX/LaTeX/BibTex is that the formatting of the output is separated from the content.
    BibTeX manages the appropriate formatting of the references based on the referencing system used. The same paper could be printed using different referencing - Harvard, Chicago, APA and etc -using the same BibTeX input file exported from Papers - with the formatting to the particular referencing system being performed by the LaTeX compilation process.
    The whole concept of using BibTeX is to take away the tedious work involved in referencing and ensuring accuracy of referencing through the application of preprepared styles.
    Hope this helps....
    Bruce Hunt
    Perth, Western Australia

  16. 137 Posted by jowens on 10 Jun, 2012 03:28 PM

    jowens's Avatar

    @bruce.hunt: I wholeheartedly agree that TeX/LaTeX/BibTex should be responsible for formatting. But I also agree with papers.michael that for titles—for specific formatting in the title itself—indicating that formatting within the BibTex entry is appropriate.

    I am looking at an upcoming paper titled "Maximizing Parallelism in the Construction of BVHs, Octrees, and k-d Trees". The "k" should be in italics (properly in math mode). No bib style will figure that out; the proper way to put this into bibtex is

    title = {Maximizing Parallelism in the Construction of BVHs, Octrees, and $k$-d Trees}

    I think if I entered the title exactly like that, including the $, it should be transferred faithfully in a BibTex export. But if I put the 'k' in italics, then Papers should export it as \emph{k}, as papers.michael suggests. Bold and italics should export as \textbf{} and \emph{}, respectively.

  17. 138 Posted by papers.michael on 13 Jun, 2012 05:01 PM

    papers.michael's Avatar

    @bruce.hunt & @jowens: Right, the issue is not referencing style, which should be handled by BibTeX, but rather domain specific formatting. In my case (systematic biology) I constantly need to italicize species names in titles, \emph{Monotropa uniflora} for example.

  18. 139 Posted by john on 10 Jul, 2012 04:11 PM

    john's Avatar

    Another +1 for automatic .bib export. Mendeley does it, and they're the big competitor, right?!

  19. 140 Posted by mulrich1 on 11 Jul, 2012 01:08 PM

    mulrich1's Avatar

    In addition, I would like to see a way to attach bibtex citation information to a single reference in my library. Often times I'll find an article or a book without corresponding metadata but can find a bibtex entry that I would like to attach to the papers reference.

  20. 141 Posted by improbable on 18 Jul, 2012 10:29 AM

    improbable's Avatar

    I'd like to second "Aldaron" in requesting what seems like a very simple thing: the various eprint / pmid -numbers really need to be exported into BibTex. For us at least this number is essential, it's the primary way anyone finds a paper you have cited.

    This shouldn't be hard, the database clearly has them. I think it would be enough to write out a bibtex field called whatever the field in the GUI is called. (If that isn't perfect for everyone then we can write ourselves scripts to tidy up, at least for now.)

  21. 142 Posted by marquett on 26 Jul, 2012 02:55 PM

    marquett's Avatar

    Hi Sergeant Papers band,

    Just joining the group, to strongly support the automatic export of BibTex entries in the .bib file attached to the .tex main file. A thrill, for sure…

    Cheers,
    Jean-Baptiste

  22. 143 Posted by junis rindermann on 27 Jul, 2012 02:14 PM

    junis rindermann's Avatar

    Hi,
    we have the same problems here in our research group: we are now using Zotero in addition to Papers because the bibtex export from Zotero is reliable enough to be used for serious work, whereas the exported bibtex library from Papers is faulty and poor.

    Please papers team, fix the BibTeX Export!!!!
    (Zotero looks quite ugly in comparison!!!)

  23. 144 Posted by Jean Francois Lalonde on 31 Aug, 2012 05:38 PM

    Jean Francois Lalonde's Avatar

    As marquett and others already pointed out, being able to export all the citations from within a manuscript to a bibtex file would be absolutely fantastic!

    Just like we can already select multiple articles, do 'right-click' -> 'copy as' -> 'bibtex record', it would be great to be able to do that on a manuscript!

    Thanks!
    JF

  24. 145 Posted by cinthia piamonteze on 21 Sep, 2012 07:52 AM

    cinthia piamonteze's Avatar

    Hi,

    I'm using papers for the first time to export my bibtex library and I like it very much.

    What I would like to have is to define which fields are exported to the bibtex file. For example, the minimal does not export the page numbers, which I find 'too' minimal, while the 'standard' export my personal annotations about the paper, which I find quite unnecessary and a bit invasive if I want to share the bib file with other people working on the same manuscript.

    So I would like to have a settings menu where I define which fields I want to have exported in the bibtex file. Or is this already possible and I did not know??

  25. 146 Posted by John Coxon on 17 Oct, 2012 09:26 PM

    John Coxon's Avatar

    I would prefer an export that used "" to denote each item rather than {} and I would also prefer one that inserted a comma after the last item (purely because I find that means I spend less time working out how I broke my BibTeX files...).

  26. 147 Posted by improbable on 22 Oct, 2012 10:01 AM

    improbable's Avatar

    I can't resist poking this thread once more... I bought papers 2 a year ago, but am still using papers 1 solely because bibtex is, for me, still broken.

    What I need is for it to export everything I can see in the window, especially, crucially, the eprint/arxiv/pmid numbers. Papers 1 does this, papers 2 does not.

    As others have said there are many formatting problems, which for now I manage to fix by running some script over it; that's not very "it just works" but probably many bibtex users can hack something together. But there's no way to recover what isn't written to the bibtex file in the first place.

  27. 148 Posted by jgc on 26 Oct, 2012 11:28 AM

    jgc's Avatar

    I come back regularly to check if there is any update on the latex citations workflow in Papers 2. I believe there is still no satisfactory solution, so I delaying the upgrade to Papers 2 again.

    My view on the subject: the insertion of \cite keys works just fine, but most people I know don't use bibtex. Is there a really simple way to make the actual system work with a simple latex bibliography ? any ideas ?

  28. 149 Posted by call girls Paris on 01 Nov, 2012 05:53 PM

    call girls Paris's Avatar

    Thank you for making the sincere attempt to speak about this on support.mekentosj.com . I feel very strong about it and would like to read more. If it's OK, as you achieve extra extensive wisdom, may you mind including extra articles similar to this one with additional info? It will be extraordinarily useful and helpful for me and my friends. regards

  29. 150 Posted by Antoine on 07 Nov, 2012 11:14 PM

    Antoine's Avatar

    My #1 suggestion would be to have a per-reference option to write my own custom bibtex entry. It would be nice, when I have an open reference, to inspect the bibtex code that is suggested by Papers2 and, if need be, I could replace it by a tweeked version that would remain attached to the reference for future use.

    Best

Reply to this discussion

Internal reply

Formatting help / Preview (switch to plain text) No formatting (switch to Markdown)

Attached Files

You can attach files up to 10MB

If you don't have an account yet, we need to confirm you're human and not a machine trying to post spam.

Keyboard shortcuts

Generic

? Show this help
ESC Blurs the current field

Comment Form

r Focus the comment reply box
^ + ↩ Submit the comment

You can use Command ⌘ instead of Control ^ on Mac