Commit Graph

228 Commits (0aa810f9c82c77dbb3da31879467b99b87db6241)

Author SHA1 Message Date
Eugen Rochko 49cc0eb3e7 Improve admin UI for custom emojis, add copy/disable/enable (#5231) 2017-10-05 23:42:05 +02:00
aschmitz 468523f4ad Non-Serial ("Snowflake") IDs (#4801)
* Use non-serial IDs

This change makes a number of nontrivial tweaks to the data model in
Mastodon:

* All IDs are now 8 byte integers (rather than mixed 4- and 8-byte)
* IDs are now assigned as:
  * Top 6 bytes: millisecond-resolution time from epoch
  * Bottom 2 bytes: serial (within the millisecond) sequence number
  * See /lib/tasks/db.rake's `define_timestamp_id` for details, but
    note that the purpose of these changes is to make it difficult to
    determine the number of objects in a table from the ID of any
    object.
* The Redis sorted set used for the feed will have values used to look
  up toots, rather than scores. This is almost always the same as the
  existing behavior, except in the case of boosted toots. This change
  was made because Redis stores scores as double-precision floats,
  which cannot store the new ID format exactly. Note that this doesn't
  cause problems with sorting/pagination, because ZREVRANGEBYSCORE
  sorts lexicographically when scores are tied. (This will still cause
  sorting issues when the ID gains a new significant digit, but that's
  extraordinarily uncommon.)

Note a couple of tradeoffs have been made in this commit:

* lib/tasks/db.rake is used to enforce many/most column constraints,
  because this commit seems likely to take a while to bring upstream.
  Enforcing a post-migrate hook is an easier way to maintain the code
  in the interim.
* Boosted toots will appear in the timeline as many times as they have
  been boosted. This is a tradeoff due to the way the feed is saved in
  Redis at the moment, but will be handled by a future commit.

This would effectively close Mastodon's #1059, as it is a
snowflake-like system of generating IDs. However, given how involved
the changes were simply within Mastodon, it may have unexpected
interactions with some clients, if they store IDs as doubles
(or as 4-byte integers). This was a problem that Twitter ran into with
their "snowflake" transition, particularly in JavaScript clients that
treated IDs as JS integers, rather than strings. It therefore would be
useful to test these changes at least in the web interface and popular
clients before pushing them to all users.

* Fix JavaScript interface with long IDs

Somewhat predictably, the JS interface handled IDs as numbers, which in
JS are IEEE double-precision floats. This loses some precision when
working with numbers as large as those generated by the new ID scheme,
so we instead handle them here as strings. This is relatively simple,
and doesn't appear to have caused any problems, but should definitely
be tested more thoroughly than the built-in tests. Several days of use
appear to support this working properly.

BREAKING CHANGE:

The major(!) change here is that IDs are now returned as strings by the
REST endpoints, rather than as integers. In practice, relatively few
changes were required to make the existing JS UI work with this change,
but it will likely hit API clients pretty hard: it's an entirely
different type to consume. (The one API client I tested, Tusky, handles
this with no problems, however.)

Twitter ran into this issue when introducing Snowflake IDs, and decided
to instead introduce an `id_str` field in JSON responses. I have opted
to *not* do that, and instead force all IDs to 64-bit integers
represented by strings in one go. (I believe Twitter exacerbated their
problem by rolling out the changes three times: once for statuses, once
for DMs, and once for user IDs, as well as by leaving an integer ID
value in JSON. As they said, "If you’re using the `id` field with JSON
in a Javascript-related language, there is a very high likelihood that
the integers will be silently munged by Javascript interpreters. In most
cases, this will result in behavior such as being unable to load or
delete a specific direct message, because the ID you're sending to the
API is different than the actual identifier associated with the
message." [1]) However, given that this is a significant change for API
users, alternatives or a transition time may be appropriate.

1: https://blog.twitter.com/developer/en_us/a/2011/direct-messages-going-snowflake-on-sep-30-2011.html

* Restructure feed pushes/unpushes

This was necessary because the previous behavior used Redis zset scores
to identify statuses, but those are IEEE double-precision floats, so we
can't actually use them to identify all 64-bit IDs. However, it leaves
the code in a much better state for refactoring reblog handling /
coalescing.

Feed-management code has been consolidated in FeedManager, including:

* BatchedRemoveStatusService no longer directly manipulates feed zsets
* RemoveStatusService no longer directly manipulates feed zsets
* PrecomputeFeedService has moved its logic to FeedManager#populate_feed

(PrecomputeFeedService largely made lots of calls to FeedManager, but
didn't follow the normal adding-to-feed process.)

This has the effect of unifying all of the feed push/unpush logic in
FeedManager, making it much more tractable to update it in the future.

Due to some additional checks that must be made during, for example,
batch status removals, some Redis pipelining has been removed. It does
not appear that this should cause significantly increased load, but if
necessary, some optimizations are possible in batch cases. These were
omitted in the pursuit of simplicity, but a batch_push and batch_unpush
would be possible in the future.

Tests were added to verify that pushes happen under expected conditions,
and to verify reblog behavior (both on pushing and unpushing). In the
case of unpushing, this includes testing behavior that currently leads
to confusion such as Mastodon's #2817, but this codifies that the
behavior is currently expected.

* Rubocop fixes

I could swear I made these changes already, but I must have lost them
somewhere along the line.

* Address review comments

This addresses the first two comments from review of this feature:

https://github.com/tootsuite/mastodon/pull/4801#discussion_r139336735
https://github.com/tootsuite/mastodon/pull/4801#discussion_r139336931

This adds an optional argument to FeedManager#key, the subtype of feed
key to generate. It also tests to ensure that FeedManager's settings are
such that reblogs won't be tracked forever.

* Hardcode IdToBigints migration columns

This addresses a comment during review:
https://github.com/tootsuite/mastodon/pull/4801#discussion_r139337452

This means we'll need to make sure that all _id columns going forward
are bigints, but that should happen automatically in most cases.

* Additional fixes for stringified IDs in JSON

These should be the last two. These were identified using eslint to try
to identify any plain casts to JavaScript numbers. (Some such casts are
legitimate, but these were not.)

Adding the following to .eslintrc.yml will identify casts to numbers:

~~~
  no-restricted-syntax:
  - warn
  - selector: UnaryExpression[operator='+'] > :not(Literal)
    message: Avoid the use of unary +
  - selector: CallExpression[callee.name='Number']
    message: Casting with Number() may coerce string IDs to numbers
~~~

The remaining three casts appear legitimate: two casts to array indices,
one in a server to turn an environment variable into a number.

* Only implement timestamp IDs for Status IDs

Per discussion in #4801, this is only being merged in for Status IDs at
this point. We do this in a migration, as there is no longer use for
a post-migration hook. We keep the initialization of the timestamp_id
function as a Rake task, as it is also needed after db:schema:load (as
db/schema.rb doesn't store Postgres functions).

* Change internal streaming payloads to stringified IDs as well

This is equivalent to 591a9af356 from
#5019, with an extra change for the addition to FeedManager#unpush.

* Ensure we have a status_id_seq sequence

Apparently this is not a given when specifying a custom ID function,
so now we ensure it gets created. This uses the generic version of this
function to more easily support adding additional tables with timestamp
IDs in the future, although it would be possible to cut this down to a
less generic version if necessary. It is only run during db:schema:load
or the relevant migration, so the overhead is extraordinarily minimal.

* Transition reblogs to new Redis format

This provides a one-way migration to transition old Redis reblog entries
into the new format, with a separate tracking entry for reblogs.

It is not invertible because doing so could (if timestamp IDs are used)
require a database query for each status in each users' feed, which is
likely to be a significant toll on major instances.

* Address review comments from @akihikodaki

No functional changes.

* Additional review changes

* Heredoc cleanup

* Run db:schema:load hooks for test in development

This matches the behavior in Rails'
ActiveRecord::Tasks::DatabaseTasks.each_current_configuration, which
would otherwise break `rake db:setup` in development.

It also moves some functionality out to a library, which will be a good
place to put additional related functionality in the near future.
2017-10-04 09:56:37 +02:00
ThibG dfaa219f88 Fix HTTP responses for salmon and ActivityPub inbox processing (#5200)
* Return sensible HTTP status for ActivityPub inbox processing

* Return sensible HTTP status for salmon slap processing

* Return additional information to debug signature verification failures
2017-10-03 23:21:19 +02:00
Eugen Rochko eb605141ff Fix #5104 - GET /api/v1/apps/verify_credentials to confirm app works (#5112) 2017-09-30 22:05:42 +02:00
Eugen Rochko 4ec1771165 Add ability to specify alternative text for media attachments (#5123)
* Fix #117 - Add ability to specify alternative text for media attachments

- POST /api/v1/media accepts `description` straight away
- PUT /api/v1/media/:id to update `description` (only for unattached ones)
- Serialized as `name` of Document object in ActivityPub
- Uploads form adjusted for better performance and description input

* Add tests

* Change undo button blend mode to difference
2017-09-28 15:31:31 +02:00
Eugen Rochko 293972f716 New API: GET /api/v1/custom_emojis to get a server's custom emojis (#5051) 2017-09-23 01:57:23 +02:00
Eugen Rochko 0ef9d45d05 Fix error when following locked accounts (#4896) 2017-09-11 23:50:37 +02:00
Eugen Rochko c450ddb613 Fix POST /api/v1/follows error when already following (#4878) 2017-09-10 15:09:06 +02:00
Eugen Rochko 9b994c4aee Fix #4794 - Fake instant follow in API response when account is believed unlocked (#4799) 2017-09-05 17:48:13 +02:00
Eugen Rochko 7dc5035031 Make PreviewCard records reuseable between statuses (#4642)
* Make PreviewCard records reuseable between statuses

**Warning!** Migration truncates preview_cards tablec

* Allow a wider thumbnail for link preview, display it in horizontal layout (#4648)

* Delete preview cards files before truncating

* Rename old table instead of truncating it

* Add mastodon:maintenance:remove_deprecated_preview_cards

* Ignore deprecated_preview_cards in schema definition

* Fix null behaviour
2017-09-01 16:20:16 +02:00
Eugen Rochko d1a78eba15 Embed modal (#4748)
* Embed modal

* Proxy OEmbed requests from web UI
2017-08-31 03:38:35 +02:00
Eugen Rochko e95bdec7c5 Update status embeds (#4742)
- Use statuses controller for embeds instead of stream entries controller
- Prefer /@:username/:id/embed URL for embeds
- Use /@:username as author_url in OEmbed
- Add follow link to embeds which opens web intent in new window
- Use redis cache in development
- Cache entire embed
2017-08-30 10:23:43 +02:00
masarakki 649a20ab46 authorize-follow-requests-after-unlocking (#4658) 2017-08-26 12:40:03 +02:00
Eugen Rochko 9caa90025f Pinned statuses (#4675)
* Pinned statuses

* yarn manage:translations
2017-08-25 01:41:18 +02:00
Eugen Rochko 74e5078795 Fix #4637 - Re-add missing doorkeeper_authorize for /api/v1/verify_credentials (#4650) 2017-08-21 00:41:08 +02:00
Yamagishi Kazutoshi 2edfdab6e6 Don't send Link header when don't know prev and next links (#4633) 2017-08-18 10:42:59 +02:00
abcang e120d09c98 Fix require_user! behavior when not logged in (#4604) 2017-08-15 14:14:12 +02:00
Eugen Rochko b7370ac8ba ActivityPub delivery (#4566)
* Deliver ActivityPub Like

* Deliver ActivityPub Undo-Like

* Deliver ActivityPub Create/Announce activities

* Deliver ActivityPub creates from mentions

* Deliver ActivityPub Block/Undo-Block

* Deliver ActivityPub Accept/Reject-Follow

* Deliver ActivityPub Undo-Follow

* Deliver ActivityPub Follow

* Deliver ActivityPub Delete activities

Incidentally fix #889

* Adjust BatchedRemoveStatusService for ActivityPub

* Add tests for ActivityPub workers

* Add tests for FollowService

* Add tests for FavouriteService, UnfollowService and PostStatusService

* Add tests for ReblogService, BlockService, UnblockService, ProcessMentionsService

* Add tests for AuthorizeFollowService, RejectFollowService, RemoveStatusService

* Add tests for BatchedRemoveStatusService

* Deliver updates to a local account to ActivityPub followers

* Minor adjustments
2017-08-13 00:44:41 +02:00
Akihiko Odaki a6ea7e282f Merge queries in Api::V1::FavouritesController (#4359) 2017-07-25 16:01:03 +02:00
Sorin Davidoi 9075c90c46 feat: Enable push subscription for mobile devices by default (#4321) 2017-07-23 23:27:23 +02:00
Eugen Rochko 1fcdaafa6f Fix webfinger retries (#4275)
* Do not raise unretryable exceptions in ResolveRemoteAccountService

* Removed fatal exceptions from ResolveRemoteAccountService

Exceptions that cannot be retried should not be raised. New exception
class for those that can be retried (Mastodon::UnexpectedResponseError)
2017-07-20 01:59:07 +02:00
Eugen Rochko 8c45cd0e36 Improve ActivityPub representations (#3844)
* Improve webfinger templates and make tests more flexible

* Clean up AS2 representation of actor

* Refactor outbox

* Create activities representation

* Add representations of followers/following collections, do not redirect /users/:username route if format is empty

* Remove unused translations

* ActivityPub endpoint for single statuses, add ActivityPub::TagManager for better
URL/URI generation

* Add ActivityPub::TagManager#to

* Represent all attachments as Document instead of Image/Video specifically
(Because for remote ones we may not know for sure)

Add mentions and hashtags representation to AP notes

* Add AP-resolvable hashtag URIs

* Use ActiveModelSerializers for ActivityPub

* Clean up unused translations

* Separate route for object and activity

* Adjust cc/to matrices

* Add to/cc to activities, ensure announce activity embeds target status and
not the wrapper status, add "id" to all collections
2017-07-15 03:01:39 +02:00
Eugen Rochko cd9b2ab2f7 Fix #2672 - Connect signed PuSH subscription requests to instance domain (#4205)
* Fix #2672 - Connect signed PuSH subscription requests to instance domain

Resolves #2739

* Fix return of locate_subscription

* Fix tests
2017-07-14 23:01:20 +02:00
unarist 1896a154f5 Fix response of unreblog/unfavourite APIs (#4204)
Both APIs process asynchronously, so reblogged/favourited fields in the response should be set to `false` manually.
2017-07-14 20:44:53 +02:00
Eugen Rochko 1618b68bfa HTTP signatures (#4146)
* Add Request class with HTTP signature generator

Spec: https://tools.ietf.org/html/draft-cavage-http-signatures-06

* Add HTTP signature verification concern

* Add test for SignatureVerification concern

* Add basic test for Request class

* Make PuSH subscribe/unsubscribe requests use new Request class

Accidentally fix lease_seconds not being set and sent properly, and
change the new minimum subscription duration to 1 day

* Make all PuSH workers use new Request class

* Make Salmon sender use new Request class

* Make FetchLinkService use new Request class

* Make FetchAtomService use the new Request class

* Make Remotable use the new Request class

* Make ResolveRemoteAccountService use the new Request class

* Add more tests

* Allow +-30 seconds window for signed request to remain valid

* Disable time window validation for signed requests, restore 7 days
as PuSH subscription duration (which was previous default due to a bug)
2017-07-14 20:41:49 +02:00
Sorin Davidoi 0c7c188c45 Web Push Notifications (#3243)
* feat: Register push subscription

* feat: Notify when mentioned

* feat: Boost, favourite, reply, follow, follow request

* feat: Notification interaction

* feat: Handle change of public key

* feat: Unsubscribe if things go wrong

* feat: Do not send normal notifications if push is enabled

* feat: Focus client if open

* refactor: Move push logic to WebPushSubscription

* feat: Better title and body

* feat: Localize messages

* chore: Fix lint errors

* feat: Settings

* refactor: Lazy load

* fix: Check if push settings exist

* feat: Device-based preferences

* refactor: Simplify logic

* refactor: Pull request feedback

* refactor: Pull request feedback

* refactor: Create /api/web/push_subscriptions endpoint

* feat: Spec PushSubscriptionController

* refactor: WebPushSubscription => Web::PushSubscription

* feat: Spec Web::PushSubscription

* feat: Display first media attachment

* feat: Support direction

* fix: Stuff broken while rebasing

* refactor: Integration with session activations

* refactor: Cleanup

* refactor: Simplify implementation

* feat: Set VAPID keys via environment

* chore: Comments

* fix: Crash when no alerts

* fix: Set VAPID keys in testing environment

* fix: Follow link

* feat: Notification actions

* fix: Delete previous subscription

* chore: Temporary logs

* refactor: Move migration to a later date

* fix: Fetch the correct session activation and misc bugs

* refactor: Move migration to a later date

* fix: Remove follow request (no notifications)

* feat: Send administrator contact to push service

* feat: Set time-to-live

* fix: Do not show sensitive images

* fix: Reducer crash in error handling

* feat: Add badge

* chore: Fix lint error

* fix: Checkbox label overlap

* fix: Check for payload support

* fix: Rename action "type" (crash in latest Chrome)

* feat: Action to expand notification

* fix: Lint errors

* fix: Unescape notification body

* fix: Do not allow boosting if the status is hidden

* feat: Add VAPID keys to the production sample environment

* fix: Strip HTML tags from status

* refactor: Better error messages

* refactor: Handle browser not implementing the VAPID protocol (Samsung Internet)

* fix: Error when target_status is nil

* fix: Handle lack of image

* fix: Delete reference to invalid subscriptions

* feat: Better error handling

* fix: Unescape HTML characters after tags are striped

* refactor: Simpify code

* fix: Modify to work with #4091

* Sort strings alphabetically

* i18n: Updated Polish translation

it annoys me that it's not fully localized :P

* refactor: Use current_session in PushSubscriptionController

* fix: Rebase mistake

* fix: Set cacheName to mastodon

* refactor: Pull request feedback

* refactor: Remove logging statements

* chore(yarn): Fix conflicts with master

* chore(yarn): Copy latest from master

* chore(yarn): Readd offline-plugin

* refactor: Use save! and update!

* refactor: Send notifications async

* fix: Allow retry when push fails

* fix: Save track for failed pushes

* fix: Minify sw.js

* fix: Remove account_id from fabricator
2017-07-13 22:15:32 +02:00
Eugen Rochko f93de3a516 Fix #3462 - Require authentication for search API (#4155)
This makes it consistent with /api/v1/accounts/search and
previous behaviour has been an oversight.
2017-07-11 17:08:26 +02:00
Yamagishi Kazutoshi f8212da329 Add attribute for default privacy to verify credentials (#4075)
* Add attribute for default privacy to verify credentials

* add raw_note

* source
2017-07-10 03:29:34 +02:00
Eugen Rochko 864e3f8d9c Replace OEmbed and initial state Rabl templates with serializers (#4110)
* Replace OEmbed Rabl template with serializer

* Replace initial state rabl with serializer
2017-07-08 14:51:05 +02:00
Eugen Rochko 8b2cad5637 Refactor JSON templates to be generated with ActiveModelSerializers instead of Rabl (#4090) 2017-07-07 04:02:06 +02:00
Eugen Rochko 42b8220632 Fix #1624 - Send e-mail notifications to admins about new reports (#3949) 2017-06-27 00:04:00 +02:00
Eugen Rochko f3be605286 Rename FollowRemoteAccountService to ResolveRemoteAccountService (#3847)
Rename Activitypub to ActivityPub
2017-06-19 01:51:04 +02:00
Matt Jankowski 2925372ff4 Move create/destroy actions for api/v1/statuses to namespace (#3678)
Each of mute, favourite, reblog has been updated to:

- Have a separate controller with just a create and destroy action
- Preserve historical route names to not break the API
- Mild refactoring to break up long methods
2017-06-10 09:39:26 +02:00
Matt Jankowski 5282ba862a Move reblogged_by and favourited_by actions out of api/v1/statuses and into unique controllers (#3646)
* Add specs for api statuses routes

* Update favourited_by and reblogged_by api routes

* Move methods into new controllers

* Use load_accounts methods to simplify index actions

* Clean up load_accounts methods

* Clean up link header generation

* Check for link headers in specs

* Remove unused actions from api/v1/statuses controller

* Remove specs for moved actions
2017-06-09 14:12:40 -04:00
Yamagishi Kazutoshi 0a0b9a271a Improve RuboCop rules (compatibility to Code Climate) (#3636)
08f8de84eb/Gemfile.lock (L38)
Code Climate is using RuboCop v0.46.0.

Change several rules to maintain compatibility.
2017-06-08 13:24:28 +02:00
Matt Jankowski 73540ffe6b Clean up for api/base controller (#3629)
* Move ApiController to Api/BaseController

* API controllers inherit from Api::BaseController

* Add coverage for various error cases in api/base controller
2017-06-07 20:09:25 +02:00
Daigo 3 Dango 2985d08951 Redirect to streaming_api_base_url (#3579)
* Redirect to streaming_api_base_url

When Rails receives a request to streaming API, it most likely
means that there is another host which is configured to respond
to it. This is to redirect clients to that host if
`STREAMING_API_BASE_URL` is set as another host.

* Use the new Ruby 1.9 hash syntax
2017-06-05 12:09:29 +02:00
Naoki Kosaka e014bf8ed0 Fix limit_param in favourites_controller.rb (#3553) 2017-06-04 14:52:26 +02:00
Matt Jankowski 5c63523972 Spec coverage and refactor for the api/v1/accounts controllers (#3451) 2017-05-31 21:36:24 +02:00
Matt Jankowski 5236a62861 Improve spec coverage and clean up api/v1/blocks controller (#3464) 2017-05-31 20:34:51 +02:00
Matt Jankowski 0f155829b7 Improve spec coverage and clean up api/v1/follow_requests controller (#3465) 2017-05-31 20:32:11 +02:00
Matt Jankowski 84dda45df9 Improve spec coverage and clean up api/v1/domain_blocks controller (#3466) 2017-05-31 20:31:14 +02:00
Matt Jankowski 9c7505489f Refactor api/v1/notifications controller (#3470) 2017-05-31 20:30:55 +02:00
Matt Jankowski 75cad1d9d6 Improve spec coverage and clean up api/v1/favourites controller (#3472) 2017-05-31 20:30:39 +02:00
Matt Jankowski bf811e4d4a Improve spec coverage and clean up api/v1/mutes controller (#3481) 2017-05-31 20:27:34 +02:00
Matt Jankowski d6774d2ca3 Refactor and spec coverage for api/v1/timelines actions (#3482) 2017-05-31 20:27:17 +02:00
Matt Jankowski 79335e46fd Refactor api/v1/apps controller (#3471) 2017-05-30 21:16:28 -04:00
Matt Jankowski 7c6e02aaf3 Refactor api/v1/reports controller (#3469) 2017-05-30 21:13:31 -04:00
Matt Jankowski 7f55430652 Refactor api/v1/search controller (#3468) 2017-05-30 21:11:54 -04:00
Matt Jankowski 8235623362 Improve spec coverage and clean up api/v1/media controller (#3467) 2017-05-30 21:11:29 -04:00